[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:07 dist-test-slave-v34r krb5kdc[394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:07 dist-test-slave-v34r krb5kdc[394](info): set up 2 sockets
Jun 20 23:28:07 dist-test-slave-v34r krb5kdc[394](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 23:28:08.970402 387 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:09 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:09.173664 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42697
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:09.512306 407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:09.512632 407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:09.512786 407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:09.520632 407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:09.520771 407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:09.520819 407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:09.520941 407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:09.521060 407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:09.531328 407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:09.532781 407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:09.534192 407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:09.548189 412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:09.548290 413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:09.548269 415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:09.549046 407 server_base.cc:1034] running on GCE node
I20260620 23:28:09.549943 407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:09.551429 407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:09.552671 407 hybrid_clock.cc:648] HybridClock initialized: now 1781998089552578 us; error 87 us; skew 500 ppm
I20260620 23:28:09.553098 407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:09 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:09.556458 407 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:09.558991 407 webserver.cc:460] Webserver started at http://127.0.96.254:43169/ using document root <none> and password file <none>
I20260620 23:28:09.559811 407 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:09.559962 407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:09.560286 407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:09.563874 407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "1fa4fa603f1744d991278bc4aff329d1"
format_stamp: "Formatted at 2026-06-20 23:28:09 on dist-test-slave-v34r"
server_key: "0e3662f99eca87c71e4f21ae94494390"
server_key_iv: "264e45556b8ab8810b6cdd10161f442f"
server_key_version: "encryptionkey@0"
I20260620 23:28:09.564702 407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "1fa4fa603f1744d991278bc4aff329d1"
format_stamp: "Formatted at 2026-06-20 23:28:09 on dist-test-slave-v34r"
server_key: "0e3662f99eca87c71e4f21ae94494390"
server_key_iv: "264e45556b8ab8810b6cdd10161f442f"
server_key_version: "encryptionkey@0"
I20260620 23:28:09.570787 407 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260620 23:28:09.574853 422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:09.576570 407 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:09.576792 407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "1fa4fa603f1744d991278bc4aff329d1"
format_stamp: "Formatted at 2026-06-20 23:28:09 on dist-test-slave-v34r"
server_key: "0e3662f99eca87c71e4f21ae94494390"
server_key_iv: "264e45556b8ab8810b6cdd10161f442f"
server_key_version: "encryptionkey@0"
I20260620 23:28:09.577071 407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:09.598151 407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:09.602535 407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:09.602836 407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:09.624358 407 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:42697
I20260620 23:28:09.624390 473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:42697 every 8 connection(s)
I20260620 23:28:09.625763 407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:09.631253 474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:09.635454 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 407
I20260620 23:28:09.635874 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:09.638797 387 external_mini_cluster.cc:1428] Setting key 241c48d3b4e0aded34650b84be6369ba
I20260620 23:28:09.647550 474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1: Bootstrap starting.
I20260620 23:28:09.651517 474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:09.652819 474 log.cc:824] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:09.656561 474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1: No bootstrap required, opened a new log
I20260620 23:28:09.665849 474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa4fa603f1744d991278bc4aff329d1" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42697 } }
I20260620 23:28:09.666385 474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:09.666535 474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa4fa603f1744d991278bc4aff329d1, State: Initialized, Role: FOLLOWER
I20260620 23:28:09.667371 474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa4fa603f1744d991278bc4aff329d1" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42697 } }
I20260620 23:28:09.667662 474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:09.667807 474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:09.668005 474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 23:28:09 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:09.672206 474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa4fa603f1744d991278bc4aff329d1" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42697 } }
I20260620 23:28:09.673132 474 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa4fa603f1744d991278bc4aff329d1; no voters:
I20260620 23:28:09.674034 474 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:09.674252 479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:09.675771 479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa4fa603f1744d991278bc4aff329d1, State: Running, Role: LEADER
I20260620 23:28:09.676559 479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa4fa603f1744d991278bc4aff329d1" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42697 } }
I20260620 23:28:09.677174 474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:09.681131 480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa4fa603f1744d991278bc4aff329d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa4fa603f1744d991278bc4aff329d1" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42697 } } }
I20260620 23:28:09.683619 480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:09.682294 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa4fa603f1744d991278bc4aff329d1. Latest consensus state: current_term: 1 leader_uuid: "1fa4fa603f1744d991278bc4aff329d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa4fa603f1744d991278bc4aff329d1" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42697 } } }
I20260620 23:28:09.684187 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:09.686393 487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:09.687724 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:09.645109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.0.96.254:42697)
0620 23:28:09.647354 (+ 2245us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:09.647370 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:09.647436 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:28:09.656798 (+ 9362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:09.656822 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:09.657365 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:09.657787 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:09.660754 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:09.662254 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:09.663927 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:09.664659 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:09.664898 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:09.673104 (+ 8206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:09.673116 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:09.673141 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:09.673195 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:09.678291 (+ 5096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:09.680037 (+ 1746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:09.680055 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:09.680069 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:09.680968 (+ 899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:09.682036 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:09.682041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:09.682045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:09.684271 (+ 2226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:09.685573 (+ 1302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:09.685893 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:28:09.686355 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":1642,"threads_started":1}
I20260620 23:28:09.703084 487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:09.716487 487 catalog_manager.cc:1347] Generated new cluster ID: 19d02ae3dbe44d22acc0c8022a3c3b81
I20260620 23:28:09.716653 487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:09.743219 487 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:09.745291 487 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:09.756152 487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa4fa603f1744d991278bc4aff329d1: Generated new TSK 0
I20260620 23:28:09.757546 487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:09.822037 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42697
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:10.237051 500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:10.237406 500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:10.237574 500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:10.244882 500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:10.245079 500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:10.245290 500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:10.255594 500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:10.257058 500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:10.258496 500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:10.274708 505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:10.275887 506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:10.276033 508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:10.276942 500 server_base.cc:1034] running on GCE node
I20260620 23:28:10.277750 500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:10.278921 500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:10.280158 500 hybrid_clock.cc:648] HybridClock initialized: now 1781998090280098 us; error 61 us; skew 500 ppm
I20260620 23:28:10.280519 500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:10 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:10.284216 500 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:10.286106 500 webserver.cc:460] Webserver started at http://127.0.96.193:42225/ using document root <none> and password file <none>
I20260620 23:28:10.286873 500 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:10.286980 500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:10.287345 500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:10.290452 500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "501d641fec99433facc4304f181b32cb"
format_stamp: "Formatted at 2026-06-20 23:28:10 on dist-test-slave-v34r"
server_key: "0cc19c54468bdb21e8446a76d682b09f"
server_key_iv: "eaef646e84adbd5689df833e3b550bd5"
server_key_version: "encryptionkey@0"
I20260620 23:28:10.291255 500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "501d641fec99433facc4304f181b32cb"
format_stamp: "Formatted at 2026-06-20 23:28:10 on dist-test-slave-v34r"
server_key: "0cc19c54468bdb21e8446a76d682b09f"
server_key_iv: "eaef646e84adbd5689df833e3b550bd5"
server_key_version: "encryptionkey@0"
I20260620 23:28:10.297132 500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:28:10.301412 515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:10.303107 500 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:10.303766 500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "501d641fec99433facc4304f181b32cb"
format_stamp: "Formatted at 2026-06-20 23:28:10 on dist-test-slave-v34r"
server_key: "0cc19c54468bdb21e8446a76d682b09f"
server_key_iv: "eaef646e84adbd5689df833e3b550bd5"
server_key_version: "encryptionkey@0"
I20260620 23:28:10.304054 500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:10.321478 500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:10.326726 500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:10.327064 500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:10.328214 500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:10.329836 500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:10.329914 500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:10.330056 500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:10.330119 500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:10.374954 500 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:37177
I20260620 23:28:10.375102 627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:37177 every 8 connection(s)
I20260620 23:28:10.376377 500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:10.381953 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 500
I20260620 23:28:10.382259 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:10.383153 387 external_mini_cluster.cc:1428] Setting key 26ebb67e6ca1f10bc26e405cfca89ab5
Jun 20 23:28:10 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:28:10.406786 630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:10.379801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52317 (local address 127.0.96.254:42697)
0620 23:28:10.380362 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:10.380367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:10.381531 (+ 1164us) server_negotiation.cc:407] Connection header received
0620 23:28:10.384760 (+ 3229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:10.384766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:10.384860 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:10.385053 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:10.387856 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.388629 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.392571 (+ 3942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.393152 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.393312 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:10.398817 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:10.398823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:10.398827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:10.398872 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:10.402192 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:10.403287 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:10.403299 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:10.403302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:10.403441 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:10.404177 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:10.404181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:10.404184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:10.404585 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:10.404722 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:10.405878 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0620 23:28:10.406217 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":313,"threads_started":1}
I20260620 23:28:10.407413 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:10.380235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42697 (local address 127.0.96.193:52317)
0620 23:28:10.381226 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:10.381341 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:10.384261 (+ 2920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:10.385409 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:10.385425 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:10.386413 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:10.387556 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:10.387595 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.389670 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.389679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:10.392306 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:10.392336 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.393473 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.393481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:10.394530 (+ 1049us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:10.395448 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:10.395477 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:10.398422 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:10.402434 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:10.402447 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:10.402499 (+ 52us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:10.403111 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:10.403672 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:10.403679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:10.403682 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:10.404014 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:10.404727 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:10.404735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:10.405347 (+ 612us) client_negotiation.cc:769] Sending connection context
0620 23:28:10.405688 (+ 341us) client_negotiation.cc:240] Negotiation successful
0620 23:28:10.406174 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":352,"threads_started":1}
I20260620 23:28:10.409461 628 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42697
I20260620 23:28:10.409960 628 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:10.411954 628 heartbeater.cc:511] Master 127.0.96.254:42697 requested a full tablet report, sending...
I20260620 23:28:10.415271 439 ts_manager.cc:194] Registered new tserver with Master: 501d641fec99433facc4304f181b32cb (127.0.96.193:37177)
I20260620 23:28:10.417672 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:52317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:10.433497 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42697
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:10.819679 634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:10.820031 634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:10.820161 634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:10.833112 634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:10.833257 634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:10.833390 634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:10.844229 634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:10.845636 634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:10.847888 634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:10.866338 640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:10.866426 642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:10.866366 639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:10.867411 634 server_base.cc:1034] running on GCE node
I20260620 23:28:10.868152 634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:10.869340 634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:10.870597 634 hybrid_clock.cc:648] HybridClock initialized: now 1781998090870513 us; error 79 us; skew 500 ppm
I20260620 23:28:10.870963 634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:10 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:10.875218 634 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:10.877033 634 webserver.cc:460] Webserver started at http://127.0.96.194:38323/ using document root <none> and password file <none>
I20260620 23:28:10.877789 634 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:10.877945 634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:10.878300 634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:10.881678 634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "506d7e3c5c174f6caf7229fd6de3ca02"
format_stamp: "Formatted at 2026-06-20 23:28:10 on dist-test-slave-v34r"
server_key: "4089e2036a34d70ac3f54837ba9c673e"
server_key_iv: "f7760892c21ba9c12360b64a6bcc3332"
server_key_version: "encryptionkey@0"
I20260620 23:28:10.883127 634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "506d7e3c5c174f6caf7229fd6de3ca02"
format_stamp: "Formatted at 2026-06-20 23:28:10 on dist-test-slave-v34r"
server_key: "4089e2036a34d70ac3f54837ba9c673e"
server_key_iv: "f7760892c21ba9c12360b64a6bcc3332"
server_key_version: "encryptionkey@0"
I20260620 23:28:10.889788 634 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 23:28:10.894498 649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:10.896414 634 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:10.896651 634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "506d7e3c5c174f6caf7229fd6de3ca02"
format_stamp: "Formatted at 2026-06-20 23:28:10 on dist-test-slave-v34r"
server_key: "4089e2036a34d70ac3f54837ba9c673e"
server_key_iv: "f7760892c21ba9c12360b64a6bcc3332"
server_key_version: "encryptionkey@0"
I20260620 23:28:10.896894 634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:10.918349 634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:10.923203 634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:10.923553 634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:10.924785 634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:10.926635 634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:10.926743 634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:10.926906 634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:10.926980 634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:10.971941 634 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:39589
I20260620 23:28:10.972049 761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:39589 every 8 connection(s)
I20260620 23:28:10.973430 634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:10.976225 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 634
I20260620 23:28:10.976476 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:10.977118 387 external_mini_cluster.cc:1428] Setting key 6aa3c829401efd20e9df621d90b64d14
Jun 20 23:28:10 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:28:11.005738 764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:10.977853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:58057 (local address 127.0.96.254:42697)
0620 23:28:10.978435 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:10.978442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:10.979733 (+ 1291us) server_negotiation.cc:407] Connection header received
0620 23:28:10.984543 (+ 4810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:10.984549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:10.984639 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:10.984793 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:10.987875 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.989144 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.991121 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.991923 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.992088 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:10.996684 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:10.996691 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:10.996697 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:10.996753 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.000272 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.001938 (+ 1666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.001943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.001946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.002260 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.002963 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.002966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.002969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.003294 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.003427 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.004735 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.004938 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":282,"threads_started":1}
I20260620 23:28:11.008236 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:10.978442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42697 (local address 127.0.96.194:58057)
0620 23:28:10.979503 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:10.979587 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:10.983985 (+ 4398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:10.985358 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:10.985409 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:10.986524 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:10.987572 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:10.987601 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.989387 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.989392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:10.990921 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:10.990938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:10.992237 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:10.992243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:10.992535 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:10.993495 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:10.993588 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:10.996330 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:11.000529 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:11.000547 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:11.000575 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:11.001715 (+ 1140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:11.002407 (+ 692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:11.002417 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:11.002425 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:11.002786 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:11.003455 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:11.003470 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:11.004173 (+ 703us) client_negotiation.cc:769] Sending connection context
0620 23:28:11.004569 (+ 396us) client_negotiation.cc:240] Negotiation successful
0620 23:28:11.005394 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":408,"threads_started":1}
I20260620 23:28:11.010439 762 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42697
I20260620 23:28:11.010893 762 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:11.011894 762 heartbeater.cc:511] Master 127.0.96.254:42697 requested a full tablet report, sending...
I20260620 23:28:11.013984 439 ts_manager.cc:194] Registered new tserver with Master: 506d7e3c5c174f6caf7229fd6de3ca02 (127.0.96.194:39589)
I20260620 23:28:11.014950 439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:58057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:11.028448 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42697
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:11.380393 768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:11.380788 768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:11.381014 768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:11.388729 768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:11.388883 768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:11.389129 768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:11.400409 768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42697
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:11.401867 768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:11.403250 768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:11.418958 773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:11.419207 774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:11.419113 776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:11.421471 768 server_base.cc:1034] running on GCE node
I20260620 23:28:11.421932 628 heartbeater.cc:503] Master 127.0.96.254:42697 was elected leader, sending a full tablet report...
I20260620 23:28:11.422268 768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:11.423502 768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:11.424857 768 hybrid_clock.cc:648] HybridClock initialized: now 1781998091424783 us; error 73 us; skew 500 ppm
I20260620 23:28:11.425331 768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:11.429364 768 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:11.431223 768 webserver.cc:460] Webserver started at http://127.0.96.195:46865/ using document root <none> and password file <none>
I20260620 23:28:11.432014 768 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:11.432127 768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:11.432498 768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:11.435907 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "f6a0e7d97a26440891228b92d35e9b05"
format_stamp: "Formatted at 2026-06-20 23:28:11 on dist-test-slave-v34r"
server_key: "6c3ddd44306e8f2763ffd34fb1feabe0"
server_key_iv: "7ae37ab516c8ee42c45354426b9befca"
server_key_version: "encryptionkey@0"
I20260620 23:28:11.436766 768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a0e7d97a26440891228b92d35e9b05"
format_stamp: "Formatted at 2026-06-20 23:28:11 on dist-test-slave-v34r"
server_key: "6c3ddd44306e8f2763ffd34fb1feabe0"
server_key_iv: "7ae37ab516c8ee42c45354426b9befca"
server_key_version: "encryptionkey@0"
I20260620 23:28:11.442970 768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 23:28:11.447188 783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:11.448737 768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:28:11.448928 768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "f6a0e7d97a26440891228b92d35e9b05"
format_stamp: "Formatted at 2026-06-20 23:28:11 on dist-test-slave-v34r"
server_key: "6c3ddd44306e8f2763ffd34fb1feabe0"
server_key_iv: "7ae37ab516c8ee42c45354426b9befca"
server_key_version: "encryptionkey@0"
I20260620 23:28:11.449205 768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:11.476145 768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:11.481437 768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:11.481753 768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:11.482839 768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:11.484273 768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:11.484352 768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:11.484516 768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:11.484572 768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:11.528631 768 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:43957
I20260620 23:28:11.528693 895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:43957 every 8 connection(s)
I20260620 23:28:11.530387 768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:11.530973 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 768
I20260620 23:28:11.531235 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:11.531920 387 external_mini_cluster.cc:1428] Setting key 4617f76e1a44a50d49d5f9659bd481ca
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:11.556246 898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.534069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:43543 (local address 127.0.96.254:42697)
0620 23:28:11.534606 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:11.534612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:11.536373 (+ 1761us) server_negotiation.cc:407] Connection header received
0620 23:28:11.538728 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:11.538733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:11.538833 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:11.538957 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:11.540787 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.541889 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.543644 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.544080 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.544229 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.548280 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:11.548287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:11.548294 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:11.548356 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.552504 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.553584 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.553600 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.553602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.553856 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.554389 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.554392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.554395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.554757 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.554893 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.555692 (+ 799us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.555916 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260620 23:28:11.557346 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.534963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42697 (local address 127.0.96.195:43543)
0620 23:28:11.536144 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:11.536219 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:11.538425 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:11.539269 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:11.539279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:11.539885 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:11.540577 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:11.540598 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.542069 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.542074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:11.543455 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:11.543468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.544402 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.544408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:11.544629 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.545494 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:11.545517 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:11.547961 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:11.552712 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:11.552721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:11.552738 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:11.553413 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:11.554003 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:11.554007 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:11.554010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:11.554259 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:11.554897 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:11.554904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:11.555310 (+ 406us) client_negotiation.cc:769] Sending connection context
0620 23:28:11.555565 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 23:28:11.556007 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":474,"threads_started":1}
I20260620 23:28:11.559258 896 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42697
I20260620 23:28:11.559621 896 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:11.560405 896 heartbeater.cc:511] Master 127.0.96.254:42697 requested a full tablet report, sending...
I20260620 23:28:11.562482 438 ts_manager.cc:194] Registered new tserver with Master: f6a0e7d97a26440891228b92d35e9b05 (127.0.96.195:43957)
I20260620 23:28:11.563877 438 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:43543
I20260620 23:28:11.563916 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:11.619886 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.600217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.0.96.193:37177)
0620 23:28:11.601177 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:11.601187 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:11.601314 (+ 127us) server_negotiation.cc:407] Connection header received
0620 23:28:11.601558 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:11.601568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:11.602107 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:11.602329 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:11.604057 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.605292 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.607165 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.607857 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.608036 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.613045 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:11.613080 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:11.613105 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:11.613147 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.616380 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.617504 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.617512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.617514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.617647 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.618177 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.618181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.618183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.618547 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.618801 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.619301 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.619539 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":371,"threads_started":1}
W20260620 23:28:11.621542 543 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58660
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:11.678126 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.661014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.0.96.193:37177)
0620 23:28:11.661322 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:11.661330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:11.661721 (+ 391us) server_negotiation.cc:407] Connection header received
0620 23:28:11.661995 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:11.662000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:11.662078 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:11.662203 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:11.664018 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.664785 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.666439 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.666955 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.667118 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.671886 (+ 4768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:11.671893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:11.671897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:11.671947 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.674916 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.675815 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.675820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.675823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.675979 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.676409 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.676413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.676415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.676712 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.677059 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.677420 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.677755 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:28:11.680153 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 500
I20260620 23:28:11.695771 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.681355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.0.96.193:37177)
0620 23:28:11.681571 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:11.681577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:11.681832 (+ 255us) server_negotiation.cc:407] Connection header received
0620 23:28:11.682065 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:11.682069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:11.682137 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:11.682249 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:11.683465 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.684258 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.685704 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.686190 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.686316 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.688896 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:11.688901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:11.688906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:11.688958 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.692763 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.693705 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.693710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.693713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.693925 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.694312 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.694315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.694317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.694629 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.694866 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.695232 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.695438 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:28:11.696733 623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:11.791211 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 500
I20260620 23:28:11.810595 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 634
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:11.828719 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.811674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.0.96.194:39589)
0620 23:28:11.812416 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:11.812425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:11.812453 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:28:11.812528 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:11.812535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:11.812749 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:11.812889 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:11.814397 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.815495 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.817029 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.817602 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.817741 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.822452 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:11.822465 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:11.822481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:11.822518 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.825616 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.826463 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.826470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.826472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.826615 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.827057 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.827060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.827062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.827415 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.827639 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.828035 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.828290 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":315,"threads_started":1}
I20260620 23:28:11.830823 757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:11.922401 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 634
I20260620 23:28:11.946993 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 768
Jun 20 23:28:11 dist-test-slave-v34r krb5kdc[394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:11.967171 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:11.948401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.96.195:43957)
0620 23:28:11.949036 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:11.949046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:11.949089 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:28:11.949160 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:11.949166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:11.949400 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:11.949558 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:11.951102 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.952172 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.954010 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:11.954767 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:11.954901 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:11.960051 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:11.960066 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:11.960083 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:11.960126 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:11.963483 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.964537 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.964545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.964548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.964695 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:11.965336 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:11.965340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:11.965342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:11.965708 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:11.965956 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:11.966447 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 23:28:11.966727 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":286,"threads_started":1}
I20260620 23:28:11.968691 891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:12.064837 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 768
I20260620 23:28:12.084692 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 407
I20260620 23:28:12.086300 469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:12.200040 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 407
2026-06-20T23:28:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4409 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:12 dist-test-slave-v34r krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:12 dist-test-slave-v34r krb5kdc[918](info): set up 2 sockets
Jun 20 23:28:12 dist-test-slave-v34r krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260620 23:28:13.325098 387 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:13 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:13.525517 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34243
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:13.846791 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:13.847100 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:13.847249 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:13.854595 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:13.854720 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:13.854794 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:13.854888 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:13.854955 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:13.864892 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:13.866287 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:13.867625 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:13.880831 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:13.880719 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:13.880874 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:13.882071 931 server_base.cc:1034] running on GCE node
I20260620 23:28:13.883169 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:13.885182 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:13.886490 931 hybrid_clock.cc:648] HybridClock initialized: now 1781998093886413 us; error 56 us; skew 500 ppm
I20260620 23:28:13.887010 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:13 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:13.895354 931 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:13.897619 931 webserver.cc:460] Webserver started at http://127.0.96.254:37729/ using document root <none> and password file <none>
I20260620 23:28:13.898535 931 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:13.898660 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:13.898998 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:13.902827 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "49f1217dc2b747399ca5c4286fe472cc"
format_stamp: "Formatted at 2026-06-20 23:28:13 on dist-test-slave-v34r"
server_key: "01e6a4d66edd6b5ad4eff93cd6011fda"
server_key_iv: "4b8857bd53529abd3c4f131c85b87333"
server_key_version: "encryptionkey@0"
I20260620 23:28:13.903736 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "49f1217dc2b747399ca5c4286fe472cc"
format_stamp: "Formatted at 2026-06-20 23:28:13 on dist-test-slave-v34r"
server_key: "01e6a4d66edd6b5ad4eff93cd6011fda"
server_key_iv: "4b8857bd53529abd3c4f131c85b87333"
server_key_version: "encryptionkey@0"
I20260620 23:28:13.910557 931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 23:28:13.915674 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:13.917793 931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:28:13.918067 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "49f1217dc2b747399ca5c4286fe472cc"
format_stamp: "Formatted at 2026-06-20 23:28:13 on dist-test-slave-v34r"
server_key: "01e6a4d66edd6b5ad4eff93cd6011fda"
server_key_iv: "4b8857bd53529abd3c4f131c85b87333"
server_key_version: "encryptionkey@0"
I20260620 23:28:13.918329 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:13.941872 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:13.947698 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:13.948196 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:13.973846 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34243
I20260620 23:28:13.973869 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34243 every 8 connection(s)
I20260620 23:28:13.975739 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:13.976555 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 931
I20260620 23:28:13.976850 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:13.977773 387 external_mini_cluster.cc:1428] Setting key 2bcc8efc44f74170fec5d316fc2b35f0
I20260620 23:28:13.982714 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:28:13 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:13.997865 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc: Bootstrap starting.
I20260620 23:28:14.002354 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:14.003700 998 log.cc:824] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:14.007783 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:13.980292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37070 (local address 127.0.96.254:34243)
0620 23:28:13.981380 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:13.981400 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:13.981458 (+ 58us) server_negotiation.cc:407] Connection header received
0620 23:28:13.983970 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:13.984001 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:13.984541 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:13.985049 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:13.987062 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:13.989179 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:13.990848 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:13.991892 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:13.992496 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:13.997329 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:13.997341 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:13.997366 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:13.997419 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:14.001721 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:14.002900 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:14.002909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:14.002919 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:14.003145 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:14.003723 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:14.003729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:14.003732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:14.004574 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:14.004835 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:14.005888 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0620 23:28:14.006385 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":444,"threads_started":1}
I20260620 23:28:14.008116 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc: No bootstrap required, opened a new log
I20260620 23:28:14.017210 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1217dc2b747399ca5c4286fe472cc" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34243 } }
I20260620 23:28:14.017617 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:14.017710 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f1217dc2b747399ca5c4286fe472cc, State: Initialized, Role: FOLLOWER
I20260620 23:28:14.018385 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1217dc2b747399ca5c4286fe472cc" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34243 } }
I20260620 23:28:14.018700 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:14.018857 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:14.019040 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:14.038125 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1217dc2b747399ca5c4286fe472cc" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34243 } }
I20260620 23:28:14.039225 998 leader_election.cc:302] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f1217dc2b747399ca5c4286fe472cc; no voters:
I20260620 23:28:14.040256 998 leader_election.cc:288] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:14.040463 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:14.041317 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [term 1 LEADER]: Becoming Leader. State: Replica: 49f1217dc2b747399ca5c4286fe472cc, State: Running, Role: LEADER
I20260620 23:28:14.042078 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1217dc2b747399ca5c4286fe472cc" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34243 } }
I20260620 23:28:14.044153 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:14.047240 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f1217dc2b747399ca5c4286fe472cc. Latest consensus state: current_term: 1 leader_uuid: "49f1217dc2b747399ca5c4286fe472cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1217dc2b747399ca5c4286fe472cc" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34243 } } }
I20260620 23:28:14.047263 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f1217dc2b747399ca5c4286fe472cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f1217dc2b747399ca5c4286fe472cc" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34243 } } }
I20260620 23:28:14.048190 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:14.048014 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:14.050726 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:14.058473 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:14.070451 1008 catalog_manager.cc:1347] Generated new cluster ID: bc237a95cf964f7bacbe41155178ac58
I20260620 23:28:14.070608 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:14.090620 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:14.092554 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:14.102941 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f1217dc2b747399ca5c4286fe472cc: Generated new TSK 0
I20260620 23:28:14.104374 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
W20260620 23:28:14.391502 387 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.96.193: real 0.269s user 0.001s sys 0.010s
I20260620 23:28:14.392632 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34243
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:14.731967 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:14.732270 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:14.732399 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:14.739774 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:14.739899 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:14.740068 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:14.750072 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:14.751483 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:14.752858 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:14.767529 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:14.767611 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:14.767663 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:14.769191 1024 server_base.cc:1034] running on GCE node
I20260620 23:28:14.770076 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:14.771363 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:14.772660 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781998094772584 us; error 66 us; skew 500 ppm
I20260620 23:28:14.773169 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:14 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:14.778456 1024 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:14.780469 1024 webserver.cc:460] Webserver started at http://127.0.96.193:34209/ using document root <none> and password file <none>
I20260620 23:28:14.781478 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:14.781610 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:14.781939 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:14.785869 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "ea458ffae6064444ab50225cb4506189"
format_stamp: "Formatted at 2026-06-20 23:28:14 on dist-test-slave-v34r"
server_key: "2a576bfa8a76edde91696bf9edbf0741"
server_key_iv: "4bf678bea995669e83a549fbddbea7ce"
server_key_version: "encryptionkey@0"
I20260620 23:28:14.786880 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "ea458ffae6064444ab50225cb4506189"
format_stamp: "Formatted at 2026-06-20 23:28:14 on dist-test-slave-v34r"
server_key: "2a576bfa8a76edde91696bf9edbf0741"
server_key_iv: "4bf678bea995669e83a549fbddbea7ce"
server_key_version: "encryptionkey@0"
I20260620 23:28:14.793933 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 23:28:14.798914 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:14.800840 1024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:14.801229 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "ea458ffae6064444ab50225cb4506189"
format_stamp: "Formatted at 2026-06-20 23:28:14 on dist-test-slave-v34r"
server_key: "2a576bfa8a76edde91696bf9edbf0741"
server_key_iv: "4bf678bea995669e83a549fbddbea7ce"
server_key_version: "encryptionkey@0"
I20260620 23:28:14.801470 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:14.821219 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:14.826885 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:14.827214 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:14.828441 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:14.830929 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:14.831038 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:14.831153 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:14.831208 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:14.871708 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:36497
I20260620 23:28:14.871793 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:36497 every 8 connection(s)
I20260620 23:28:14.873891 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:14.877238 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1024
I20260620 23:28:14.877453 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:14.878263 387 external_mini_cluster.cc:1428] Setting key 007d41d0a05cc7f4bb4341d3c7952d6b
Jun 20 23:28:14 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:28:14.903147 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:14.877178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52901 (local address 127.0.96.254:34243)
0620 23:28:14.879764 (+ 2586us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:14.879770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:14.879795 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:28:14.882863 (+ 3068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:14.882868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:14.882953 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:14.883077 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:14.886261 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:14.887286 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:14.889373 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:14.890220 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:14.890410 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:14.894957 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:14.894967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:14.894974 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:14.895037 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:14.898867 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:14.899964 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:14.899970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:14.899974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:14.900121 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:14.900676 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:14.900681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:14.900689 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:14.901127 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:14.901327 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:14.902491 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0620 23:28:14.902776 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":2348,"threads_started":1}
I20260620 23:28:14.904233 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:14.877773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34243 (local address 127.0.96.193:52901)
0620 23:28:14.879203 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:14.879313 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:14.882281 (+ 2968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:14.883528 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:14.883548 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:14.884699 (+ 1151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:14.885941 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:14.885983 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:14.887535 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:14.887540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:14.889162 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:14.889181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:14.890549 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:14.890556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:14.890858 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:14.891746 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:14.891777 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:14.894561 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:14.899119 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:14.899131 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:14.899154 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:14.899798 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:14.900257 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:14.900267 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:14.900270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:14.900543 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:14.901391 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:14.901407 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:14.901941 (+ 534us) client_negotiation.cc:769] Sending connection context
0620 23:28:14.902306 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 23:28:14.902938 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":549,"threads_started":1}
I20260620 23:28:14.906383 1152 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34243
I20260620 23:28:14.906744 1152 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:14.907693 1152 heartbeater.cc:511] Master 127.0.96.254:34243 requested a full tablet report, sending...
I20260620 23:28:14.910317 963 ts_manager.cc:194] Registered new tserver with Master: ea458ffae6064444ab50225cb4506189 (127.0.96.193:36497)
I20260620 23:28:14.911942 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:52901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:14.930210 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34243
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:15.331591 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:15.331887 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:15.332036 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:15.339430 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:15.339551 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:15.339738 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:15.349931 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:15.351280 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:15.352692 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:15.367547 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:15.367498 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:15.367594 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:15.368389 1158 server_base.cc:1034] running on GCE node
I20260620 23:28:15.369132 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:15.370278 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:15.371658 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781998095371586 us; error 59 us; skew 500 ppm
I20260620 23:28:15.372077 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:15 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:15.375588 1158 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:15.377336 1158 webserver.cc:460] Webserver started at http://127.0.96.194:44127/ using document root <none> and password file <none>
I20260620 23:28:15.378065 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:15.378173 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:15.378455 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:15.381577 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "6d66bc9829d4421d8873c61a64498ca5"
format_stamp: "Formatted at 2026-06-20 23:28:15 on dist-test-slave-v34r"
server_key: "d21101861b033f47d1ea4a23aa166fad"
server_key_iv: "bd8979eb70a759f2de4fe991b8027703"
server_key_version: "encryptionkey@0"
I20260620 23:28:15.382367 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "6d66bc9829d4421d8873c61a64498ca5"
format_stamp: "Formatted at 2026-06-20 23:28:15 on dist-test-slave-v34r"
server_key: "d21101861b033f47d1ea4a23aa166fad"
server_key_iv: "bd8979eb70a759f2de4fe991b8027703"
server_key_version: "encryptionkey@0"
I20260620 23:28:15.388360 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 23:28:15.392452 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:15.394320 1158 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 23:28:15.394522 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "6d66bc9829d4421d8873c61a64498ca5"
format_stamp: "Formatted at 2026-06-20 23:28:15 on dist-test-slave-v34r"
server_key: "d21101861b033f47d1ea4a23aa166fad"
server_key_iv: "bd8979eb70a759f2de4fe991b8027703"
server_key_version: "encryptionkey@0"
I20260620 23:28:15.394726 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:15.426404 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:15.431098 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:15.431417 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:15.432533 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:15.434185 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:15.434264 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:15.434393 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:15.434464 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:15.473182 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:33323
I20260620 23:28:15.473281 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:33323 every 8 connection(s)
I20260620 23:28:15.474732 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:15.484570 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1158
I20260620 23:28:15.484807 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:15.485399 387 external_mini_cluster.cc:1428] Setting key f83b2bac3129156dfbc06009803c4587
Jun 20 23:28:15 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:15.497012 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:15.478124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:52637 (local address 127.0.96.254:34243)
0620 23:28:15.478642 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:15.478648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:15.480146 (+ 1498us) server_negotiation.cc:407] Connection header received
0620 23:28:15.482302 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:15.482308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:15.482396 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:15.482524 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:15.484312 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:15.485181 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:15.486978 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:15.487491 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:15.487628 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:15.491260 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:15.491265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:15.491268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:15.491301 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:15.493969 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:15.494711 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:15.494718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:15.494722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:15.494840 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:15.495269 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:15.495274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:15.495277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:15.495549 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:15.495708 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:15.496410 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 23:28:15.496618 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":279,"threads_started":1}
I20260620 23:28:15.498173 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:15.479044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34243 (local address 127.0.96.194:52637)
0620 23:28:15.479945 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:15.479996 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:15.481955 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:15.482792 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:15.482802 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:15.483420 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:15.484106 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:15.484126 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:15.485339 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:15.485343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:15.486839 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:15.486851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:15.487736 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:15.487752 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:15.487996 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:15.488703 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:15.488728 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:15.490997 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:15.494130 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:15.494139 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:15.494159 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:15.494589 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:15.494946 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:15.494952 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:15.494956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:15.495146 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:15.495676 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:15.495684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:15.496071 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 23:28:15.496339 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 23:28:15.496891 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":356,"threads_started":1}
I20260620 23:28:15.500123 1286 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34243
I20260620 23:28:15.500456 1286 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:15.501256 1286 heartbeater.cc:511] Master 127.0.96.254:34243 requested a full tablet report, sending...
I20260620 23:28:15.502982 963 ts_manager.cc:194] Registered new tserver with Master: 6d66bc9829d4421d8873c61a64498ca5 (127.0.96.194:33323)
I20260620 23:28:15.503830 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:52637
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:15.533488 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34243
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:28:15.915424 1152 heartbeater.cc:503] Master 127.0.96.254:34243 was elected leader, sending a full tablet report...
W20260620 23:28:15.947355 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:15.947817 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:15.948005 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:15.956719 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:15.956955 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:15.957173 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:15.968966 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34243
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:15.970870 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:15.972740 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:15.989648 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:15.989529 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:15.989583 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:15.990562 1292 server_base.cc:1034] running on GCE node
I20260620 23:28:15.991278 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:15.992365 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:15.993626 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781998095993557 us; error 64 us; skew 500 ppm
I20260620 23:28:15.994024 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:15 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:15.997349 1292 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:15.999233 1292 webserver.cc:460] Webserver started at http://127.0.96.195:43427/ using document root <none> and password file <none>
I20260620 23:28:16.000233 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:16.000368 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:16.000725 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:16.003947 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf"
format_stamp: "Formatted at 2026-06-20 23:28:15 on dist-test-slave-v34r"
server_key: "0e08995399e10dddfe96b54d2e5cc776"
server_key_iv: "34e1fbb097e4585304aa63a4c3752f7b"
server_key_version: "encryptionkey@0"
I20260620 23:28:16.004762 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf"
format_stamp: "Formatted at 2026-06-20 23:28:15 on dist-test-slave-v34r"
server_key: "0e08995399e10dddfe96b54d2e5cc776"
server_key_iv: "34e1fbb097e4585304aa63a4c3752f7b"
server_key_version: "encryptionkey@0"
I20260620 23:28:16.010728 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 23:28:16.014941 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:16.016580 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:28:16.016785 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf"
format_stamp: "Formatted at 2026-06-20 23:28:15 on dist-test-slave-v34r"
server_key: "0e08995399e10dddfe96b54d2e5cc776"
server_key_iv: "34e1fbb097e4585304aa63a4c3752f7b"
server_key_version: "encryptionkey@0"
I20260620 23:28:16.017056 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:16.037087 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:16.042153 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:16.042493 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:16.043565 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:16.045233 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:16.045316 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:16.045462 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:16.045513 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:16.083997 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:33539
I20260620 23:28:16.084188 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:33539 every 8 connection(s)
I20260620 23:28:16.085469 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:16.095570 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1292
I20260620 23:28:16.095847 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:16.096354 387 external_mini_cluster.cc:1428] Setting key 2422b379b3cb27f7d4bc9f670476ed5c
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:16.108552 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.088650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:49977 (local address 127.0.96.254:34243)
0620 23:28:16.089115 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.089120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.090406 (+ 1286us) server_negotiation.cc:407] Connection header received
0620 23:28:16.092677 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.092684 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.092801 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.092924 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.094673 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.095518 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.097473 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.097945 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.098100 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.102444 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.102449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.102452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.102483 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.105330 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.106172 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.106177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.106180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.106315 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.106737 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.106742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.106745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.107080 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.107243 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.107941 (+ 698us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.108152 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":237,"threads_started":1}
I20260620 23:28:16.109630 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.089336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34243 (local address 127.0.96.195:49977)
0620 23:28:16.090220 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:16.090272 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:16.092333 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:16.093247 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:16.093257 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:16.093834 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:16.094483 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.094501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.095693 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.095700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.097318 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.097330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.098249 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.098260 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.098502 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.099264 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:16.099294 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:16.102176 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:16.105488 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:16.105499 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:16.105522 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:16.106039 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:16.106425 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:16.106430 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:16.106434 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:16.106617 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:16.107214 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:16.107223 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:16.107570 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 23:28:16.107846 (+ 276us) client_negotiation.cc:240] Negotiation successful
0620 23:28:16.108390 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":343,"threads_started":1}
I20260620 23:28:16.111521 1420 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34243
I20260620 23:28:16.111881 1420 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:16.112772 1420 heartbeater.cc:511] Master 127.0.96.254:34243 requested a full tablet report, sending...
I20260620 23:28:16.115139 963 ts_manager.cc:194] Registered new tserver with Master: 0b5d92e7fdc24cbd8efadd0c505c2ecf (127.0.96.195:33539)
I20260620 23:28:16.115998 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:49977
I20260620 23:28:16.126592 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:28:16.146764 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.133617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.96.254:34243)
0620 23:28:16.133878 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.133886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.134433 (+ 547us) server_negotiation.cc:407] Connection header received
0620 23:28:16.134653 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.134658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.134723 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.134828 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.136412 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.137168 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.138415 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.138802 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.138936 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.141390 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.141395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.141398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.141428 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.144072 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.144702 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.144709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.144713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.144832 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.145350 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.145355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.145358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.145676 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.145921 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.146243 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.146463 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 23:28:16.154574 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:16.162446 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:16.204373 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.177395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.0.96.193:36497)
0620 23:28:16.184687 (+ 7292us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.184702 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.194073 (+ 9371us) server_negotiation.cc:407] Connection header received
0620 23:28:16.194535 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.194550 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.194955 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.195228 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:16.196442 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.198115 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.201572 (+ 3457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.203042 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.203197 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.203442 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.203835 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.204016 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7022,"thread_start_us":6824,"threads_started":1}
I20260620 23:28:16.204452 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.176528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36497 (local address 127.0.0.1:35910)
0620 23:28:16.193843 (+ 17315us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:16.193874 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:16.194259 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:16.195684 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:16.195692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:16.195721 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:16.196239 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.196256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.198282 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.198288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.201353 (+ 3065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.201366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.203284 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.203290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.203523 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.203589 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 23:28:16.203759 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:28:16.204020 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17103,"thread_start_us":19320,"threads_started":1}
I20260620 23:28:16.205440 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.176329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:33323 (local address 127.0.0.1:49764)
0620 23:28:16.186316 (+ 9987us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:16.186405 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:16.186779 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:16.196755 (+ 9976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:16.196761 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:16.196790 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:16.197251 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.197276 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.199285 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.199295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.202842 (+ 3547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.202862 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.204627 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.204633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.204767 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.204795 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:28:16.204848 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 23:28:16.204957 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9503,"thread_start_us":17075,"threads_started":1}
I20260620 23:28:16.205888 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.176546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.0.96.194:33323)
0620 23:28:16.195974 (+ 19428us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.195985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.196023 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:28:16.196162 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.196168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.196474 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.196767 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:16.197556 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.199125 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.203051 (+ 3926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.204374 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.205096 (+ 722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.205287 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.205374 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.205546 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19197,"thread_start_us":18978,"threads_started":1}
I20260620 23:28:16.207338 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.177889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58428 (local address 127.0.96.195:33539)
0620 23:28:16.192832 (+ 14943us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.192847 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.196399 (+ 3552us) server_negotiation.cc:407] Connection header received
0620 23:28:16.196530 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.196540 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.196770 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.197013 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:16.197877 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.199655 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.203285 (+ 3630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.204891 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.205044 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.205384 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.206722 (+ 1338us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.206959 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14653,"thread_start_us":14960,"threads_started":1}
I20260620 23:28:16.207983 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.177038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:33539 (local address 127.0.0.1:58428)
0620 23:28:16.196217 (+ 19179us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:16.196238 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:16.196392 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:16.197140 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:16.197146 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:16.197181 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:16.197654 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.197667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.199832 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.199839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.203071 (+ 3232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.203084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.205626 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.205632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.206460 (+ 828us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.206497 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:28:16.207621 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0620 23:28:16.207752 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19022,"thread_start_us":8493,"threads_started":1}
I20260620 23:28:16.210711 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c11631be11b3485ebd69c44b3e0a2a40 (DEFAULT_TABLE table=test-table [id=eaf4e694f50546f1b10f1bbb699e057b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:16.210712 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c11631be11b3485ebd69c44b3e0a2a40 (DEFAULT_TABLE table=test-table [id=eaf4e694f50546f1b10f1bbb699e057b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:16.212016 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c11631be11b3485ebd69c44b3e0a2a40 (DEFAULT_TABLE table=test-table [id=eaf4e694f50546f1b10f1bbb699e057b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:16.212600 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11631be11b3485ebd69c44b3e0a2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:16.212615 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11631be11b3485ebd69c44b3e0a2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:16.213548 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11631be11b3485ebd69c44b3e0a2a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:16.229267 1439 tablet_bootstrap.cc:492] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: Bootstrap starting.
I20260620 23:28:16.233975 1441 tablet_bootstrap.cc:492] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: Bootstrap starting.
I20260620 23:28:16.235532 1439 tablet_bootstrap.cc:654] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:16.237838 1440 tablet_bootstrap.cc:492] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: Bootstrap starting.
I20260620 23:28:16.238179 1439 log.cc:824] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:16.239671 1441 tablet_bootstrap.cc:654] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:16.241487 1441 log.cc:824] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:16.241772 1440 tablet_bootstrap.cc:654] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:16.242645 1439 tablet_bootstrap.cc:492] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: No bootstrap required, opened a new log
I20260620 23:28:16.243041 1439 ts_tablet_manager.cc:1397] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.003s
I20260620 23:28:16.243235 1440 log.cc:824] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:16.245298 1441 tablet_bootstrap.cc:492] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: No bootstrap required, opened a new log
I20260620 23:28:16.245684 1441 ts_tablet_manager.cc:1397] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260620 23:28:16.247553 1440 tablet_bootstrap.cc:492] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: No bootstrap required, opened a new log
I20260620 23:28:16.247836 1440 ts_tablet_manager.cc:1397] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260620 23:28:16.252852 1439 raft_consensus.cc:348] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.253429 1439 raft_consensus.cc:374] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:16.253547 1439 raft_consensus.cc:729] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5d92e7fdc24cbd8efadd0c505c2ecf, State: Initialized, Role: FOLLOWER
I20260620 23:28:16.254330 1439 consensus_queue.cc:260] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.255702 1420 heartbeater.cc:503] Master 127.0.96.254:34243 was elected leader, sending a full tablet report...
I20260620 23:28:16.256568 1439 ts_tablet_manager.cc:1428] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260620 23:28:16.257728 1441 raft_consensus.cc:348] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.258229 1441 raft_consensus.cc:374] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:16.258354 1441 raft_consensus.cc:729] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea458ffae6064444ab50225cb4506189, State: Initialized, Role: FOLLOWER
I20260620 23:28:16.258859 1440 raft_consensus.cc:348] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.259259 1440 raft_consensus.cc:374] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:16.259343 1440 raft_consensus.cc:729] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d66bc9829d4421d8873c61a64498ca5, State: Initialized, Role: FOLLOWER
I20260620 23:28:16.259189 1441 consensus_queue.cc:260] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.260106 1440 consensus_queue.cc:260] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.261744 1286 heartbeater.cc:503] Master 127.0.96.254:34243 was elected leader, sending a full tablet report...
I20260620 23:28:16.262725 1440 ts_tablet_manager.cc:1428] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: Time spent starting tablet: real 0.015s user 0.000s sys 0.014s
I20260620 23:28:16.265564 1441 ts_tablet_manager.cc:1428] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260620 23:28:16.323629 1447 raft_consensus.cc:482] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:16.324015 1447 raft_consensus.cc:504] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.326216 1447 leader_election.cc:288] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea458ffae6064444ab50225cb4506189 (127.0.96.193:36497), 0b5d92e7fdc24cbd8efadd0c505c2ecf (127.0.96.195:33539)
I20260620 23:28:16.333998 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.327032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:41031 (local address 127.0.96.195:33539)
0620 23:28:16.327289 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.327298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.328508 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 23:28:16.328777 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.328782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.328864 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.328977 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:16.329537 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.330312 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.332204 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.333030 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.333227 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.333316 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.333582 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.333707 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:28:16.334009 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.326618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:43789 (local address 127.0.96.193:36497)
0620 23:28:16.326890 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.326896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.327992 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 23:28:16.328252 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.328258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.328335 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.328434 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:16.329099 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.329936 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.332039 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.332891 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.333143 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.333236 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.333537 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.333688 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 23:28:16.334229 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.326859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:33539 (local address 127.0.96.194:41031)
0620 23:28:16.328395 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:16.328412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:16.328616 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:16.329022 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:16.329027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:16.329052 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:16.329397 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.329406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.330458 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.330463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.332051 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.332063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.333261 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.333267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.333428 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.333453 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 23:28:16.333559 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:28:16.333714 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1399,"mutex_wait_us":1,"spinlock_wait_cycles":1792,"thread_start_us":1122,"threads_started":1}
I20260620 23:28:16.334712 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.326770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36497 (local address 127.0.96.194:43789)
0620 23:28:16.327807 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:16.327860 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:16.328084 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:16.328465 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:16.328471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:16.328497 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:16.328899 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.328910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.330063 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.330070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.331868 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:16.331882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.333163 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.333170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:16.333355 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.333382 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:28:16.333494 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:28:16.333683 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":78,"spinlock_wait_cycles":1280,"thread_start_us":652,"threads_started":1}
I20260620 23:28:16.335640 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11631be11b3485ebd69c44b3e0a2a40" candidate_uuid: "6d66bc9829d4421d8873c61a64498ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" is_pre_election: true
I20260620 23:28:16.336445 1375 raft_consensus.cc:2454] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d66bc9829d4421d8873c61a64498ca5 in term 0.
I20260620 23:28:16.336452 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11631be11b3485ebd69c44b3e0a2a40" candidate_uuid: "6d66bc9829d4421d8873c61a64498ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea458ffae6064444ab50225cb4506189" is_pre_election: true
I20260620 23:28:16.337213 1107 raft_consensus.cc:2454] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d66bc9829d4421d8873c61a64498ca5 in term 0.
W20260620 23:28:16.337287 1421 tablet.cc:2369] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:28:16.337316 1176 leader_election.cc:302] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b5d92e7fdc24cbd8efadd0c505c2ecf, 6d66bc9829d4421d8873c61a64498ca5; no voters:
I20260620 23:28:16.338295 1447 raft_consensus.cc:2781] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:16.338477 1447 raft_consensus.cc:482] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:16.338595 1447 raft_consensus.cc:3037] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:16.341099 1447 raft_consensus.cc:504] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.341813 1447 leader_election.cc:288] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [CANDIDATE]: Term 1 election: Requested vote from peers ea458ffae6064444ab50225cb4506189 (127.0.96.193:36497), 0b5d92e7fdc24cbd8efadd0c505c2ecf (127.0.96.195:33539)
I20260620 23:28:16.342492 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11631be11b3485ebd69c44b3e0a2a40" candidate_uuid: "6d66bc9829d4421d8873c61a64498ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea458ffae6064444ab50225cb4506189"
I20260620 23:28:16.342489 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c11631be11b3485ebd69c44b3e0a2a40" candidate_uuid: "6d66bc9829d4421d8873c61a64498ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf"
I20260620 23:28:16.342759 1375 raft_consensus.cc:3037] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:16.342759 1107 raft_consensus.cc:3037] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:16.346498 1107 raft_consensus.cc:2454] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d66bc9829d4421d8873c61a64498ca5 in term 1.
I20260620 23:28:16.346498 1375 raft_consensus.cc:2454] T c11631be11b3485ebd69c44b3e0a2a40 P 0b5d92e7fdc24cbd8efadd0c505c2ecf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d66bc9829d4421d8873c61a64498ca5 in term 1.
I20260620 23:28:16.347193 1175 leader_election.cc:302] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d66bc9829d4421d8873c61a64498ca5, ea458ffae6064444ab50225cb4506189; no voters:
I20260620 23:28:16.347606 1447 raft_consensus.cc:2781] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:16.348193 1447 raft_consensus.cc:686] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 6d66bc9829d4421d8873c61a64498ca5, State: Running, Role: LEADER
I20260620 23:28:16.348810 1447 consensus_queue.cc:237] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } }
I20260620 23:28:16.354035 963 catalog_manager.cc:5495] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d66bc9829d4421d8873c61a64498ca5 (127.0.96.194). New cstate: current_term: 1 leader_uuid: "6d66bc9829d4421d8873c61a64498ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d66bc9829d4421d8873c61a64498ca5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 33323 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea458ffae6064444ab50225cb4506189" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b5d92e7fdc24cbd8efadd0c505c2ecf" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33539 } health_report { overall_health: UNKNOWN } } }
W20260620 23:28:16.377362 1153 tablet.cc:2369] T c11631be11b3485ebd69c44b3e0a2a40 P ea458ffae6064444ab50225cb4506189: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:16.409382 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.392794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.0.96.193:36497)
0620 23:28:16.393108 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.393117 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.393658 (+ 541us) server_negotiation.cc:407] Connection header received
0620 23:28:16.393881 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.393886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.393965 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.394074 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.395555 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.396314 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.397546 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.397994 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.398125 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.402951 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.402965 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.402984 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.403017 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.406362 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.407366 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.407373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.407375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.407519 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.407938 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.407942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.407944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.408306 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.408529 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.408862 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.409089 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998096, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:16.462365 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.445158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.0.96.193:36497)
0620 23:28:16.445398 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.445409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.445956 (+ 547us) server_negotiation.cc:407] Connection header received
0620 23:28:16.446226 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.446231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.446307 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.446415 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.447977 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.448796 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.450172 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.450793 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.450943 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.455719 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.455726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.455730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.455779 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.459296 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.460084 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.460090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.460093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.460270 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.460732 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.460737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.460740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.461174 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.461427 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.461784 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.462019 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 23:28:16.477180 1287 tablet.cc:2369] T c11631be11b3485ebd69c44b3e0a2a40 P 6d66bc9829d4421d8873c61a64498ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:16.515353 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.498857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.96.193:36497)
0620 23:28:16.499171 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.499181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.499630 (+ 449us) server_negotiation.cc:407] Connection header received
0620 23:28:16.499823 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.499827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.499952 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.500054 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.501945 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.502664 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.503987 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.504453 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.504557 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.509856 (+ 5299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.509862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.509866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.509902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.512487 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.513306 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.513310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.513312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.513425 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.513785 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.513788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.513790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.514094 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.514398 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.514741 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.514973 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:28:16.517731 1067 tablet_service.cc:2908] Rejecting scan request for tablet c11631be11b3485ebd69c44b3e0a2a40: Uninitialized: clean time has not yet been initialized
I20260620 23:28:16.521262 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1024
I20260620 23:28:16.535394 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.522804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.0.96.193:36497)
0620 23:28:16.522997 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.523001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.523561 (+ 560us) server_negotiation.cc:407] Connection header received
0620 23:28:16.523770 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.523774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.523852 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.523944 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.525389 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.526044 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.527371 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.527788 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.527898 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.530496 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.530500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.530504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.530533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.532911 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.533670 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.533675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.533677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.533793 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.534155 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.534158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.534160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.534423 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.534623 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.534967 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.535125 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:28:16.536221 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:16.635160 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1024
I20260620 23:28:16.655121 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1158
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:16.670933 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.656065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.0.96.194:33323)
0620 23:28:16.656285 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.656290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.656352 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:28:16.656424 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.656428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.656511 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.656606 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.657920 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.658691 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.659970 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.660410 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.660517 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.664860 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.664876 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.664893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.664940 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.667854 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.668807 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.668814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.668817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.669039 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.669516 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.669520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.669522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.669894 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.670127 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.670482 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.670668 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:28:16.671830 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:16.765477 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1158
I20260620 23:28:16.782848 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1292
Jun 20 23:28:16 dist-test-slave-v34r krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:16.799479 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:16.784016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.0.96.195:33539)
0620 23:28:16.784211 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:16.784218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:16.784293 (+ 75us) server_negotiation.cc:407] Connection header received
0620 23:28:16.784579 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:16.784587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:16.784701 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:16.784839 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:16.786479 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.787289 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.788545 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:16.788936 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:16.789103 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:16.793576 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:16.793592 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:16.793609 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:16.793646 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:16.796480 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.797246 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.797254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.797256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.797395 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:16.797815 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:16.797825 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:16.797827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:16.798182 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:16.798520 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:16.798861 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:28:16.799098 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:28:16.800390 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:16.898419 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1292
I20260620 23:28:16.915247 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 931
I20260620 23:28:16.916635 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:17.025210 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 931
2026-06-20T23:28:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4818 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:17 dist-test-slave-v34r krb5kdc[1469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:17 dist-test-slave-v34r krb5kdc[1469](info): set up 2 sockets
Jun 20 23:28:17 dist-test-slave-v34r krb5kdc[1469](info): commencing operation
krb5kdc: starting...
W20260620 23:28:18.137274 387 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:18 dist-test-slave-v34r krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:18.346297 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42049
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:18.699824 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:18.700126 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:18.700258 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:18.707836 1482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:18.707954 1482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:28:18.707996 1482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:18.708122 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:18.708201 1482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:18.708318 1482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:18.718506 1482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:18.719856 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:18.721285 1482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:18.733946 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:18.734160 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:18.734186 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:18.735483 1482 server_base.cc:1034] running on GCE node
I20260620 23:28:18.736580 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:18.738487 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:18.739840 1482 hybrid_clock.cc:648] HybridClock initialized: now 1781998098739680 us; error 73 us; skew 500 ppm
I20260620 23:28:18.740342 1482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:18 dist-test-slave-v34r krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:18.744143 1482 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:18.746083 1482 webserver.cc:460] Webserver started at http://127.0.96.254:38495/ using document root <none> and password file <none>
I20260620 23:28:18.747014 1482 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:18.747118 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:18.747529 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:18.751389 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "9ed5faeb2f904149beac974967d40fb5"
format_stamp: "Formatted at 2026-06-20 23:28:18 on dist-test-slave-v34r"
server_key: "235ae41f873ab6447171cbcb5835eb63"
server_key_iv: "458096dab1356f0c422524f71309802d"
server_key_version: "encryptionkey@0"
I20260620 23:28:18.752477 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "9ed5faeb2f904149beac974967d40fb5"
format_stamp: "Formatted at 2026-06-20 23:28:18 on dist-test-slave-v34r"
server_key: "235ae41f873ab6447171cbcb5835eb63"
server_key_iv: "458096dab1356f0c422524f71309802d"
server_key_version: "encryptionkey@0"
I20260620 23:28:18.759294 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260620 23:28:18.763827 1497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:18.765863 1482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:28:18.766111 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "9ed5faeb2f904149beac974967d40fb5"
format_stamp: "Formatted at 2026-06-20 23:28:18 on dist-test-slave-v34r"
server_key: "235ae41f873ab6447171cbcb5835eb63"
server_key_iv: "458096dab1356f0c422524f71309802d"
server_key_version: "encryptionkey@0"
I20260620 23:28:18.766397 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:18.796662 1482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:18.801103 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:18.801395 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:18.825856 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:42049
I20260620 23:28:18.825958 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:42049 every 8 connection(s)
I20260620 23:28:18.827309 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:18.828367 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1482
I20260620 23:28:18.828660 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:18.829607 387 external_mini_cluster.cc:1428] Setting key 0970ce35ad109c6e5b5be1e1721fc149
I20260620 23:28:18.831847 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:18.843732 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: Bootstrap starting.
Jun 20 23:28:18 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:18.846802 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:18.847819 1559 log.cc:824] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:18.850716 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: No bootstrap required, opened a new log
I20260620 23:28:18.857376 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:18.831988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.96.254:42049)
0620 23:28:18.832910 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:18.832928 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:18.833053 (+ 125us) server_negotiation.cc:407] Connection header received
0620 23:28:18.835067 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:18.835098 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:18.835676 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:18.836215 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:18.838073 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:18.839709 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:18.841233 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:18.842178 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:18.842423 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:18.847532 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:18.847542 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:18.847570 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:18.847628 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:18.851711 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:18.852686 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:18.852695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:18.852704 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:18.852941 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:18.853492 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:18.853497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:18.853499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:18.854251 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:18.854656 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:18.855219 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 23:28:18.855825 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":409,"threads_started":1}
I20260620 23:28:18.859174 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed5faeb2f904149beac974967d40fb5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42049 } }
I20260620 23:28:18.859596 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:18.859740 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed5faeb2f904149beac974967d40fb5, State: Initialized, Role: FOLLOWER
I20260620 23:28:18.860343 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed5faeb2f904149beac974967d40fb5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42049 } }
I20260620 23:28:18.860596 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:18.860742 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:18.860931 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:18.863240 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed5faeb2f904149beac974967d40fb5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42049 } }
I20260620 23:28:18.863762 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed5faeb2f904149beac974967d40fb5; no voters:
I20260620 23:28:18.864441 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:18.864779 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:18.865921 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed5faeb2f904149beac974967d40fb5, State: Running, Role: LEADER
I20260620 23:28:18.866617 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed5faeb2f904149beac974967d40fb5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42049 } }
I20260620 23:28:18.866864 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:18.870285 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed5faeb2f904149beac974967d40fb5. Latest consensus state: current_term: 1 leader_uuid: "9ed5faeb2f904149beac974967d40fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed5faeb2f904149beac974967d40fb5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42049 } } }
I20260620 23:28:18.870814 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:18.871543 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ed5faeb2f904149beac974967d40fb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed5faeb2f904149beac974967d40fb5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42049 } } }
I20260620 23:28:18.871945 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:18.871953 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:18.878775 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:18.891042 1571 catalog_manager.cc:1347] Generated new cluster ID: 924ce307c66a41b1a3897a1f9bcc165e
I20260620 23:28:18.891198 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:18.906176 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:18.907655 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:18.917610 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: Generated new TSK 0
I20260620 23:28:18.918715 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:18.977583 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42049
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:19.390909 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:19.391206 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:19.391309 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:19.398561 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:19.398680 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:19.398764 1585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:28:19.398837 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:19.409511 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:19.410941 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:19.412285 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:19.427737 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:19.427673 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:19.428448 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:19.429080 1585 server_base.cc:1034] running on GCE node
I20260620 23:28:19.429855 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:19.431064 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:19.432314 1585 hybrid_clock.cc:648] HybridClock initialized: now 1781998099432239 us; error 47 us; skew 500 ppm
I20260620 23:28:19.432715 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:19 dist-test-slave-v34r krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:19.440898 1585 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:19.443127 1585 webserver.cc:460] Webserver started at http://127.0.96.193:33303/ using document root <none> and password file <none>
I20260620 23:28:19.444118 1585 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:19.444298 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:19.444751 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:19.448761 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "c744552582d24932be55167ff0a0f199"
format_stamp: "Formatted at 2026-06-20 23:28:19 on dist-test-slave-v34r"
server_key: "cedf26a714c4d75bd17e7fdc10a9d951"
server_key_iv: "426b1deb654d55856b7b00f12d3d988c"
server_key_version: "encryptionkey@0"
I20260620 23:28:19.450066 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "c744552582d24932be55167ff0a0f199"
format_stamp: "Formatted at 2026-06-20 23:28:19 on dist-test-slave-v34r"
server_key: "cedf26a714c4d75bd17e7fdc10a9d951"
server_key_iv: "426b1deb654d55856b7b00f12d3d988c"
server_key_version: "encryptionkey@0"
I20260620 23:28:19.457746 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 23:28:19.462477 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:19.464399 1585 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:19.464687 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "c744552582d24932be55167ff0a0f199"
format_stamp: "Formatted at 2026-06-20 23:28:19 on dist-test-slave-v34r"
server_key: "cedf26a714c4d75bd17e7fdc10a9d951"
server_key_iv: "426b1deb654d55856b7b00f12d3d988c"
server_key_version: "encryptionkey@0"
I20260620 23:28:19.465029 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:19.480041 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:19.485915 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:19.486281 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:19.489717 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:19.489850 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:19.490021 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:19.490095 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:28:19 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:19.521435 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:19.490632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:45397 (local address 127.0.96.254:42049)
0620 23:28:19.491139 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:19.491144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:19.492894 (+ 1750us) server_negotiation.cc:407] Connection header received
0620 23:28:19.495238 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:19.495243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:19.495336 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:19.495465 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:19.498206 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.499627 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.503740 (+ 4113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.504312 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.504544 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:19.511369 (+ 6825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:19.511381 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:19.511390 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:19.511462 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:19.515799 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:19.517035 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:19.517043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:19.517048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:19.517220 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:19.518338 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:19.518349 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:19.518356 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:19.518909 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:19.519133 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:19.520576 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0620 23:28:19.520911 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":281,"threads_started":1}
I20260620 23:28:19.523909 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:19.491625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.96.193:45397)
0620 23:28:19.492680 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:19.492752 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:19.494924 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:19.495819 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:19.495838 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:19.496808 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:19.497874 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:19.497922 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.500023 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.500031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:19.503462 (+ 3431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:19.503496 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.504769 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.504783 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:19.505288 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:19.506354 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:19.506403 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:19.510900 (+ 4497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:19.516045 (+ 5145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:19.516058 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:19.516076 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:19.516846 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:19.517499 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:19.517509 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:19.517517 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:19.518076 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:19.519112 (+ 1036us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:19.519124 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:19.519784 (+ 660us) client_negotiation.cc:769] Sending connection context
0620 23:28:19.520313 (+ 529us) client_negotiation.cc:240] Negotiation successful
0620 23:28:19.521350 (+ 1037us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":470,"threads_started":1}
I20260620 23:28:19.543151 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:46305
I20260620 23:28:19.543272 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:46305 every 8 connection(s)
I20260620 23:28:19.544837 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:19.547519 1721 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42049
I20260620 23:28:19.547816 1721 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:19.548632 1721 heartbeater.cc:511] Master 127.0.96.254:42049 requested a full tablet report, sending...
I20260620 23:28:19.553391 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1585
I20260620 23:28:19.553602 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:19.554730 387 external_mini_cluster.cc:1428] Setting key e4f50c8d3eeefd71fb5455f63a83f37b
I20260620 23:28:19.557847 1514 ts_manager.cc:194] Registered new tserver with Master: c744552582d24932be55167ff0a0f199 (127.0.96.193:46305)
I20260620 23:28:19.560122 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:45397
I20260620 23:28:19.561782 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:19.541622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:45553 (local address 127.0.96.254:42049)
0620 23:28:19.542062 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:19.542068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:19.545923 (+ 3855us) server_negotiation.cc:407] Connection header received
0620 23:28:19.546631 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:19.546639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:19.546740 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:19.546810 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:19.548350 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.549344 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.550705 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.551183 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.551338 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:19.553913 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:19.553918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:19.553922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:19.553959 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:19.558506 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:19.559376 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:19.559384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:19.559388 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:19.559558 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:19.559984 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:19.559997 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:19.559999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:19.560387 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:19.560532 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:19.561078 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 23:28:19.561359 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260620 23:28:19.561908 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:19.541577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.96.193:45553)
0620 23:28:19.545689 (+ 4112us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:19.545730 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:19.546019 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:19.547110 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:19.547115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:19.547775 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:19.548188 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:19.548200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.549497 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.549504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:19.550546 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:19.550558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:19.551393 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:19.551400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:19.551597 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:19.552554 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:19.552574 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:19.553709 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:19.558708 (+ 4999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:19.558714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:19.558718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:19.559218 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:19.559696 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:19.559702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:19.559706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:19.559854 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:19.560590 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:19.560594 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:19.560773 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 23:28:19.560930 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:28:19.561382 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3844,"thread_start_us":3624,"threads_started":1}
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:19.619971 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42049
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:19.972285 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:19.972580 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:19.972735 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:19.980391 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:19.980511 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:19.980623 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:28:19.980736 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:19.990772 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:19.992090 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:19.993525 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:20.009198 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:20.009291 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:20.009161 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:20.015611 1725 server_base.cc:1034] running on GCE node
I20260620 23:28:20.016460 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:20.017715 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:20.019213 1725 hybrid_clock.cc:648] HybridClock initialized: now 1781998100019118 us; error 49 us; skew 500 ppm
I20260620 23:28:20.019801 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:20 dist-test-slave-v34r krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:20.025673 1725 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:20.028050 1725 webserver.cc:460] Webserver started at http://127.0.96.194:33033/ using document root <none> and password file <none>
I20260620 23:28:20.029187 1725 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:20.029346 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:20.029767 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:20.033553 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "db4ac78a531e4fd998d5fb3dca6a784b"
format_stamp: "Formatted at 2026-06-20 23:28:20 on dist-test-slave-v34r"
server_key: "9dfd58be08739e992c92c2bf64aa9720"
server_key_iv: "2926fb42dbecbc8fc56c26510d57d155"
server_key_version: "encryptionkey@0"
I20260620 23:28:20.034600 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "db4ac78a531e4fd998d5fb3dca6a784b"
format_stamp: "Formatted at 2026-06-20 23:28:20 on dist-test-slave-v34r"
server_key: "9dfd58be08739e992c92c2bf64aa9720"
server_key_iv: "2926fb42dbecbc8fc56c26510d57d155"
server_key_version: "encryptionkey@0"
I20260620 23:28:20.041559 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 23:28:20.046370 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:20.048326 1725 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:20.048566 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "db4ac78a531e4fd998d5fb3dca6a784b"
format_stamp: "Formatted at 2026-06-20 23:28:20 on dist-test-slave-v34r"
server_key: "9dfd58be08739e992c92c2bf64aa9720"
server_key_iv: "2926fb42dbecbc8fc56c26510d57d155"
server_key_version: "encryptionkey@0"
I20260620 23:28:20.048857 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:20.064015 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:20.069906 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:20.070254 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:20.074074 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:20.074222 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:20.074391 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:20.074700 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 20 23:28:20 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:20.103544 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.074623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:41253 (local address 127.0.96.254:42049)
0620 23:28:20.075182 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.075191 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.077645 (+ 2454us) server_negotiation.cc:407] Connection header received
0620 23:28:20.080202 (+ 2557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.080209 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.080314 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.080454 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:20.083432 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.084631 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.088738 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.089556 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.089730 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.095293 (+ 5563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:20.095299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:20.095303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:20.095339 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:20.098799 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.099961 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.099965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.099968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.100084 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.100774 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.100783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.100787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.101419 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:20.101688 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.102844 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.103103 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":278,"threads_started":1}
I20260620 23:28:20.104732 1750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.075986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.96.194:41253)
0620 23:28:20.077439 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.077494 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.079852 (+ 2358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.080851 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.080866 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.081982 (+ 1116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:20.083124 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.083146 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.084928 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.084939 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.088408 (+ 3469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.088439 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.089950 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.089963 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.090422 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.091549 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:20.091581 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:20.094992 (+ 3411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:20.099013 (+ 4021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.099025 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.099051 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.099798 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.100218 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.100223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.100227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.100521 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.101631 (+ 1110us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:20.101645 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:20.102276 (+ 631us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.102720 (+ 444us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.103339 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":638,"threads_started":1}
I20260620 23:28:20.132532 1813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.112869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.96.194:35663)
0620 23:28:20.114073 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.114118 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.114406 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.114897 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.114906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.115881 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:20.116661 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.116682 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.118075 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.118082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.120099 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.120123 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.121306 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.121319 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.121709 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.122921 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:20.122962 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:20.124633 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:20.128623 (+ 3990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.128630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.128634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.129657 (+ 1023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.130205 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.130210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.130214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.130427 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.131475 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:20.131481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:20.131676 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.131812 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.132099 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":733,"threads_started":1}
I20260620 23:28:20.133036 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.112830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:35663 (local address 127.0.96.254:42049)
0620 23:28:20.113195 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.113203 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.114300 (+ 1097us) server_negotiation.cc:407] Connection header received
0620 23:28:20.114592 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.114601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.114696 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.114801 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:20.116921 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.117909 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.120286 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.120898 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.121080 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.124969 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:20.124975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:20.125048 (+ 73us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:20.125098 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:20.128415 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.129898 (+ 1483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.129905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.129911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.130069 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.130597 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.130606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.130613 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.131275 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:20.131488 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.131992 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.132464 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:28:20.139879 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:32975
I20260620 23:28:20.139886 1860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:32975 every 8 connection(s)
I20260620 23:28:20.141556 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:20.143181 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1725
I20260620 23:28:20.143385 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:20.143889 387 external_mini_cluster.cc:1428] Setting key b7d772942259b4b306b8e8954e80bd0a
I20260620 23:28:20.144276 1861 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42049
I20260620 23:28:20.144565 1861 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:20.145370 1861 heartbeater.cc:511] Master 127.0.96.254:42049 requested a full tablet report, sending...
I20260620 23:28:20.147027 1514 ts_manager.cc:194] Registered new tserver with Master: db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975)
I20260620 23:28:20.148092 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:41253
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:20.193717 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42049
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:20.546888 1865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:20.547240 1865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:20.547451 1865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:20.555076 1865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:20.555215 1865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:20.555392 1865 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:28:20.555521 1865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:20.564854 1721 heartbeater.cc:503] Master 127.0.96.254:42049 was elected leader, sending a full tablet report...
I20260620 23:28:20.566246 1865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:42049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:20.567838 1865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:20.569346 1865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:20.585000 1871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:20.584918 1873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:20.585080 1870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:20.590735 1865 server_base.cc:1034] running on GCE node
I20260620 23:28:20.591451 1865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:20.592478 1865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:20.593734 1865 hybrid_clock.cc:648] HybridClock initialized: now 1781998100593661 us; error 58 us; skew 500 ppm
I20260620 23:28:20.594043 1865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:20 dist-test-slave-v34r krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:20.597541 1865 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:20.599227 1865 webserver.cc:460] Webserver started at http://127.0.96.195:46673/ using document root <none> and password file <none>
I20260620 23:28:20.599984 1865 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:20.600113 1865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:20.600445 1865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:20.603585 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "bc34ab1c25e84cadb13fabcf0524d3f6"
format_stamp: "Formatted at 2026-06-20 23:28:20 on dist-test-slave-v34r"
server_key: "34d0ee7f9618d8db5eff08bcc854a96d"
server_key_iv: "287182399cc58da74b8c7b3d4a70ebb0"
server_key_version: "encryptionkey@0"
I20260620 23:28:20.604398 1865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "bc34ab1c25e84cadb13fabcf0524d3f6"
format_stamp: "Formatted at 2026-06-20 23:28:20 on dist-test-slave-v34r"
server_key: "34d0ee7f9618d8db5eff08bcc854a96d"
server_key_iv: "287182399cc58da74b8c7b3d4a70ebb0"
server_key_version: "encryptionkey@0"
I20260620 23:28:20.610172 1865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260620 23:28:20.614387 1880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:20.616151 1865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:28:20.616382 1865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "bc34ab1c25e84cadb13fabcf0524d3f6"
format_stamp: "Formatted at 2026-06-20 23:28:20 on dist-test-slave-v34r"
server_key: "34d0ee7f9618d8db5eff08bcc854a96d"
server_key_iv: "287182399cc58da74b8c7b3d4a70ebb0"
server_key_version: "encryptionkey@0"
I20260620 23:28:20.616634 1865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:20.663316 1865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:20.668433 1865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:20.668764 1865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:20.672065 1865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:20.672176 1865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:20.672401 1865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:20.672485 1865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:28:20 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:20.701992 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.673549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:46845 (local address 127.0.96.254:42049)
0620 23:28:20.674127 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.674135 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.676191 (+ 2056us) server_negotiation.cc:407] Connection header received
0620 23:28:20.679361 (+ 3170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.679368 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.679493 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.679644 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:20.682122 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.683380 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.685683 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.686362 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.686558 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.693657 (+ 7099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:20.693670 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:20.693679 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:20.693749 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:20.697900 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.698982 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.698987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.698990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.699141 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.699657 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.699661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.699664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.699971 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:20.700146 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.701114 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.701389 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":271,"threads_started":1}
I20260620 23:28:20.704087 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.674496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.96.195:46845)
0620 23:28:20.675940 (+ 1444us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.675999 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.678873 (+ 2874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.679953 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.679964 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.680633 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:20.681820 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.681862 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.683582 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.683589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.685430 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.685456 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.686977 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.686983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.687525 (+ 542us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.688707 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:20.688735 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:20.693191 (+ 4456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:20.698140 (+ 4949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.698152 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.698184 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.698800 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.699267 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.699272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.699276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.699515 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.700128 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:20.700139 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:20.700637 (+ 498us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.700957 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.701824 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"thread_start_us":550,"threads_started":1}
I20260620 23:28:20.730681 1865 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:44773
I20260620 23:28:20.730798 2000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:44773 every 8 connection(s)
I20260620 23:28:20.732291 1865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:20.735278 2001 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42049
I20260620 23:28:20.735623 2001 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:20.736265 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.716161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.96.195:56395)
0620 23:28:20.717718 (+ 1557us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.717749 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.718082 (+ 333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.718857 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.718864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.719857 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:20.720356 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.720369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.722179 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.722192 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.724779 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.724807 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.726265 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.726275 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.726514 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.727460 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:20.727488 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:20.728823 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:20.732942 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.732947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.732956 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.733622 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.734155 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:20.734158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:20.734160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:20.734326 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:20.735139 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:20.735144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:20.735450 (+ 306us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.735575 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.735891 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"thread_start_us":1035,"threads_started":1}
I20260620 23:28:20.736435 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.716045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:56395 (local address 127.0.96.254:42049)
0620 23:28:20.716262 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.716267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.717906 (+ 1639us) server_negotiation.cc:407] Connection header received
0620 23:28:20.718319 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.718328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.718444 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.718533 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:20.720538 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.721848 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.725072 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.725939 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.726131 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.729200 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:20.729210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:20.729217 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:20.729275 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:20.732709 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.733844 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.733852 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.733858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.734032 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.734506 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.734516 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.734520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.734948 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:20.735176 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.735757 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.736046 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:28:20.736774 2001 heartbeater.cc:511] Master 127.0.96.254:42049 requested a full tablet report, sending...
I20260620 23:28:20.737480 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 1865
I20260620 23:28:20.737648 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:20.738178 387 external_mini_cluster.cc:1428] Setting key 1efac455bc32f2f174d52296e27e8347
I20260620 23:28:20.738617 1514 ts_manager.cc:194] Registered new tserver with Master: bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195:44773)
I20260620 23:28:20.739813 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:46845
I20260620 23:28:20.741966 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:28:20 dist-test-slave-v34r krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:28:20 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:20.794967 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.779212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.96.254:42049)
0620 23:28:20.779492 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.779499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.779873 (+ 374us) server_negotiation.cc:407] Connection header received
0620 23:28:20.780056 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.780060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.780146 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.780259 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:20.781980 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.782740 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.784285 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.784689 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.784819 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.789242 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:20.789247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:20.789251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:20.789289 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:20.792174 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.792935 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.792939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.792942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.793182 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:20.793576 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:20.793580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:20.793582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:20.793869 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:20.794069 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.794394 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.794571 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:28:20.800073 1514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:20.809548 1514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:20.850613 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.824234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.0.96.193:46305)
0620 23:28:20.837952 (+ 13718us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.837965 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.839213 (+ 1248us) server_negotiation.cc:407] Connection header received
0620 23:28:20.840352 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.840366 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.840700 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.840904 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:20.842059 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.843680 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.846963 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.848789 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.848907 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.849218 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.849975 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.850232 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13522,"thread_start_us":13368,"threads_started":1}
I20260620 23:28:20.851823 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.823925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:46305 (local address 127.0.0.1:59724)
0620 23:28:20.838912 (+ 14987us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.839000 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.839477 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.841306 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.841315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.841349 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:20.841799 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.841812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.843879 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.843885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.846752 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.846774 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.849439 (+ 2665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.849446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.849677 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.849730 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.851004 (+ 1274us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.851325 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14539,"thread_start_us":21189,"threads_started":1}
I20260620 23:28:20.856958 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.825643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.96.194:32975)
0620 23:28:20.840173 (+ 14530us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.840181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.845718 (+ 5537us) server_negotiation.cc:407] Connection header received
0620 23:28:20.846093 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.846105 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.846422 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.846685 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:20.848134 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.849857 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.853773 (+ 3916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.855421 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.855545 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.855771 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.856472 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.856659 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14343,"thread_start_us":14214,"threads_started":1}
I20260620 23:28:20.857395 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.824114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:44773 (local address 127.0.0.1:35698)
0620 23:28:20.843876 (+ 19762us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.843912 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.846020 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.847230 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.847239 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.847267 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:20.847809 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.847822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.849908 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.849913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.854019 (+ 4106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.854037 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.856431 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.856439 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.856591 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.856617 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.856729 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.856906 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19499,"mutex_wait_us":777,"thread_start_us":20050,"threads_started":1}
I20260620 23:28:20.857406 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.824405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35698 (local address 127.0.96.195:44773)
0620 23:28:20.837132 (+ 12727us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:20.837148 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:20.844114 (+ 6966us) server_negotiation.cc:407] Connection header received
0620 23:28:20.846247 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:20.846257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:20.846592 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:20.846811 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:20.848055 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.849721 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.854227 (+ 4506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.855822 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.855990 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.856274 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:20.856794 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 23:28:20.857025 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12376,"thread_start_us":12782,"threads_started":1}
I20260620 23:28:20.857082 1651 tablet_service.cc:1467] Processing CreateTablet for tablet e4181ac3118948a18e694eb5629227db (DEFAULT_TABLE table=test-table [id=a25719c4f2d04760809ad041f931c95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:20.858421 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:20.825266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:32975 (local address 127.0.0.1:46298)
0620 23:28:20.845555 (+ 20289us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:20.845578 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:20.845844 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:20.846955 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:20.846961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:20.846993 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:20.847542 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.847554 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.850262 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.850269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.853536 (+ 3267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:20.853555 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:20.856017 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:20.856024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:20.856211 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:20.856257 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:28:20.857797 (+ 1540us) client_negotiation.cc:240] Negotiation successful
0620 23:28:20.857991 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20120,"mutex_wait_us":1,"thread_start_us":13003,"threads_started":1}
I20260620 23:28:20.858984 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4181ac3118948a18e694eb5629227db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:20.862211 1931 tablet_service.cc:1467] Processing CreateTablet for tablet e4181ac3118948a18e694eb5629227db (DEFAULT_TABLE table=test-table [id=a25719c4f2d04760809ad041f931c95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:20.862732 1791 tablet_service.cc:1467] Processing CreateTablet for tablet e4181ac3118948a18e694eb5629227db (DEFAULT_TABLE table=test-table [id=a25719c4f2d04760809ad041f931c95b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:20.864079 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4181ac3118948a18e694eb5629227db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:20.864480 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4181ac3118948a18e694eb5629227db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:20.876737 2020 tablet_bootstrap.cc:492] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Bootstrap starting.
I20260620 23:28:20.881515 2020 tablet_bootstrap.cc:654] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:20.883178 2020 log.cc:824] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:20.890384 2022 tablet_bootstrap.cc:492] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Bootstrap starting.
I20260620 23:28:20.892707 2023 tablet_bootstrap.cc:492] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Bootstrap starting.
I20260620 23:28:20.897069 2023 tablet_bootstrap.cc:654] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:20.897572 2022 tablet_bootstrap.cc:654] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:20.898627 2023 log.cc:824] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:20.898699 2020 tablet_bootstrap.cc:492] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: No bootstrap required, opened a new log
I20260620 23:28:20.899008 2020 ts_tablet_manager.cc:1397] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20260620 23:28:20.899178 2022 log.cc:824] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:20.904840 2022 tablet_bootstrap.cc:492] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: No bootstrap required, opened a new log
I20260620 23:28:20.905262 2022 ts_tablet_manager.cc:1397] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260620 23:28:20.905928 2023 tablet_bootstrap.cc:492] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: No bootstrap required, opened a new log
I20260620 23:28:20.906239 2023 ts_tablet_manager.cc:1397] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260620 23:28:20.911085 2020 raft_consensus.cc:348] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:20.911576 2020 raft_consensus.cc:374] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:20.911684 2020 raft_consensus.cc:729] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c744552582d24932be55167ff0a0f199, State: Initialized, Role: FOLLOWER
I20260620 23:28:20.912509 2020 consensus_queue.cc:260] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:20.915320 2022 raft_consensus.cc:348] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:20.915907 2022 raft_consensus.cc:374] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:20.916013 2022 raft_consensus.cc:729] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc34ab1c25e84cadb13fabcf0524d3f6, State: Initialized, Role: FOLLOWER
I20260620 23:28:20.916558 2020 ts_tablet_manager.cc:1428] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Time spent starting tablet: real 0.017s user 0.017s sys 0.001s
I20260620 23:28:20.916884 2022 consensus_queue.cc:260] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:20.918720 2023 raft_consensus.cc:348] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:20.919142 2023 raft_consensus.cc:374] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:20.919273 2023 raft_consensus.cc:729] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4ac78a531e4fd998d5fb3dca6a784b, State: Initialized, Role: FOLLOWER
I20260620 23:28:20.920049 2023 consensus_queue.cc:260] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:20.920502 2022 ts_tablet_manager.cc:1428] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Time spent starting tablet: real 0.015s user 0.012s sys 0.001s
I20260620 23:28:20.920928 2001 heartbeater.cc:503] Master 127.0.96.254:42049 was elected leader, sending a full tablet report...
I20260620 23:28:20.922313 1861 heartbeater.cc:503] Master 127.0.96.254:42049 was elected leader, sending a full tablet report...
I20260620 23:28:20.923182 2023 ts_tablet_manager.cc:1428] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260620 23:28:20.984486 2002 tablet.cc:2369] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:28:21.048066 1722 tablet.cc:2369] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:28:21.144315 1862 tablet.cc:2369] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:28:21.277844 2027 raft_consensus.cc:482] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:21.278228 2027 raft_consensus.cc:504] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:21.280205 2027 leader_election.cc:288] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c744552582d24932be55167ff0a0f199 (127.0.96.193:46305), db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975)
I20260620 23:28:21.290628 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.280864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:43977 (local address 127.0.96.194:32975)
0620 23:28:21.281157 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.281164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.281588 (+ 424us) server_negotiation.cc:407] Connection header received
0620 23:28:21.281830 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.281835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.281926 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.282041 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:21.282895 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.284027 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.288029 (+ 4002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.289391 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.289560 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.289644 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.290239 (+ 595us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.290361 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:28:21.290628 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.280762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:37257 (local address 127.0.96.193:46305)
0620 23:28:21.281098 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.281107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.281699 (+ 592us) server_negotiation.cc:407] Connection header received
0620 23:28:21.281829 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.281836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.281926 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.282041 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:21.282895 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.284027 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.288028 (+ 4001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.289355 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.289560 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.289641 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.290210 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.290348 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260620 23:28:21.290903 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.280632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:46305 (local address 127.0.96.195:37257)
0620 23:28:21.281560 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.281586 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.281684 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.282081 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.282087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.282113 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:21.282612 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.282632 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.284222 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.284228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.287741 (+ 3513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.287765 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.289652 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.289656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.289959 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.290036 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.290157 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.290383 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":338,"thread_start_us":550,"threads_started":1}
I20260620 23:28:21.291463 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.280632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:32975 (local address 127.0.96.195:43977)
0620 23:28:21.281420 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.281445 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.281673 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.282081 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.282087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.282179 (+ 92us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:21.282654 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.282666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.284223 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.284228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.287742 (+ 3514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.287764 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.289628 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.289641 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.289966 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.290038 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.290157 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.290383 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":356,"spinlock_wait_cycles":38016,"thread_start_us":442,"threads_started":1}
I20260620 23:28:21.292346 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4181ac3118948a18e694eb5629227db" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c744552582d24932be55167ff0a0f199" is_pre_election: true
I20260620 23:28:21.292735 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4181ac3118948a18e694eb5629227db" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" is_pre_election: true
I20260620 23:28:21.292943 1671 raft_consensus.cc:2454] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 0.
I20260620 23:28:21.293339 1816 raft_consensus.cc:2454] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 0.
I20260620 23:28:21.293774 1884 leader_election.cc:302] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc34ab1c25e84cadb13fabcf0524d3f6, c744552582d24932be55167ff0a0f199; no voters:
I20260620 23:28:21.294260 2027 raft_consensus.cc:2781] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:21.294445 2027 raft_consensus.cc:482] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:21.294615 2027 raft_consensus.cc:3037] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.297638 2027 raft_consensus.cc:504] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:21.298457 2027 leader_election.cc:288] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 election: Requested vote from peers c744552582d24932be55167ff0a0f199 (127.0.96.193:46305), db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975)
I20260620 23:28:21.299108 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4181ac3118948a18e694eb5629227db" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c744552582d24932be55167ff0a0f199"
I20260620 23:28:21.299209 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4181ac3118948a18e694eb5629227db" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4ac78a531e4fd998d5fb3dca6a784b"
I20260620 23:28:21.299371 1671 raft_consensus.cc:3037] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.299515 1816 raft_consensus.cc:3037] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.301860 1816 raft_consensus.cc:2454] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 1.
I20260620 23:28:21.302151 1671 raft_consensus.cc:2454] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 1.
I20260620 23:28:21.302407 1882 leader_election.cc:302] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc34ab1c25e84cadb13fabcf0524d3f6, db4ac78a531e4fd998d5fb3dca6a784b; no voters:
I20260620 23:28:21.302811 2027 raft_consensus.cc:2781] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:21.303305 2027 raft_consensus.cc:686] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 LEADER]: Becoming Leader. State: Replica: bc34ab1c25e84cadb13fabcf0524d3f6, State: Running, Role: LEADER
I20260620 23:28:21.303892 2027 consensus_queue.cc:237] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } }
I20260620 23:28:21.309809 1512 catalog_manager.cc:5495] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } health_report { overall_health: UNKNOWN } } }
I20260620 23:28:21.440083 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.433324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35702 (local address 127.0.96.195:44773)
0620 23:28:21.433834 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.433839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.434434 (+ 595us) server_negotiation.cc:407] Connection header received
0620 23:28:21.434768 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.434772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.434871 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.435039 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:21.435536 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.436221 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.437815 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.438221 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.438348 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.438874 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.439434 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.439553 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.439737 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.439830 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":277,"threads_started":1}
I20260620 23:28:21.449520 1671 raft_consensus.cc:1261] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Refusing update from remote peer bc34ab1c25e84cadb13fabcf0524d3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:21.449523 1816 raft_consensus.cc:1261] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Refusing update from remote peer bc34ab1c25e84cadb13fabcf0524d3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:21.450789 2031 consensus_queue.cc:1035] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:21.451251 2027 consensus_queue.cc:1035] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:21.463318 2036 mvcc.cc:205] Tried to move back new op lower bound from 7299064223523860480 to 7299064222945816576. Current Snapshot: MvccSnapshot[applied={T|T < 7299064223523860480}]
I20260620 23:28:21.490975 1513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:21.491369 1513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:21.495970 1513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: Sending DeleteTablet for 3 replicas of tablet e4181ac3118948a18e694eb5629227db
I20260620 23:28:21.497534 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet e4181ac3118948a18e694eb5629227db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:21 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:35698
I20260620 23:28:21.497862 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet e4181ac3118948a18e694eb5629227db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:21 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:59724
I20260620 23:28:21.497815 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet e4181ac3118948a18e694eb5629227db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:21 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:46298
I20260620 23:28:21.502656 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:21.504395 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:21.508896 2050 tablet_replica.cc:330] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: stopping tablet replica
I20260620 23:28:21.517551 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 380d34ca5b6547a6be822d986afc308e (DEFAULT_TABLE table=test-table [id=d635074e1f3b436f8df6f2fd842c87ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:21.517899 1930 tablet_service.cc:1467] Processing CreateTablet for tablet 380d34ca5b6547a6be822d986afc308e (DEFAULT_TABLE table=test-table [id=d635074e1f3b436f8df6f2fd842c87ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:21.518163 1791 tablet_service.cc:1467] Processing CreateTablet for tablet 380d34ca5b6547a6be822d986afc308e (DEFAULT_TABLE table=test-table [id=d635074e1f3b436f8df6f2fd842c87ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:21.518332 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380d34ca5b6547a6be822d986afc308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:21.518587 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380d34ca5b6547a6be822d986afc308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:21.518644 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380d34ca5b6547a6be822d986afc308e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:21.528683 2050 raft_consensus.cc:2229] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:21.529960 2049 tablet_replica.cc:330] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: stopping tablet replica
I20260620 23:28:21.530220 2053 tablet_bootstrap.cc:492] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Bootstrap starting.
I20260620 23:28:21.530745 2049 raft_consensus.cc:2229] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:21.531203 2052 tablet_bootstrap.cc:492] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Bootstrap starting.
I20260620 23:28:21.531430 2049 raft_consensus.cc:2258] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:21.532326 2050 raft_consensus.cc:2258] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:21.532486 2053 tablet_bootstrap.cc:654] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:21.535522 2048 tablet_replica.cc:330] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: stopping tablet replica
I20260620 23:28:21.535571 2050 ts_tablet_manager.cc:1905] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:21.536650 2049 ts_tablet_manager.cc:1905] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:21.537106 2052 tablet_bootstrap.cc:654] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:21.537083 2048 raft_consensus.cc:2229] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:21.541076 2048 raft_consensus.cc:2258] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:21.541530 2053 tablet_bootstrap.cc:492] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: No bootstrap required, opened a new log
I20260620 23:28:21.542140 2053 ts_tablet_manager.cc:1397] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260620 23:28:21.543015 2053 raft_consensus.cc:348] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.543259 2053 raft_consensus.cc:374] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:21.543339 2053 raft_consensus.cc:729] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c744552582d24932be55167ff0a0f199, State: Initialized, Role: FOLLOWER
I20260620 23:28:21.543599 2053 consensus_queue.cc:260] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.545944 2048 ts_tablet_manager.cc:1905] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:21.547700 2051 tablet_bootstrap.cc:492] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Bootstrap starting.
I20260620 23:28:21.554313 2051 tablet_bootstrap.cc:654] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:21.555601 2053 ts_tablet_manager.cc:1428] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20260620 23:28:21.554946 2048 ts_tablet_manager.cc:1918] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:21.556284 2048 log.cc:1196] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/e4181ac3118948a18e694eb5629227db
I20260620 23:28:21.557068 2048 ts_tablet_manager.cc:1939] T e4181ac3118948a18e694eb5629227db P bc34ab1c25e84cadb13fabcf0524d3f6: Deleting consensus metadata
I20260620 23:28:21.557770 2052 tablet_bootstrap.cc:492] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: No bootstrap required, opened a new log
I20260620 23:28:21.558097 2052 ts_tablet_manager.cc:1397] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20260620 23:28:21.559126 2052 raft_consensus.cc:348] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.559489 2052 raft_consensus.cc:374] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:21.559641 2052 raft_consensus.cc:729] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4ac78a531e4fd998d5fb3dca6a784b, State: Initialized, Role: FOLLOWER
I20260620 23:28:21.559979 2052 consensus_queue.cc:260] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.560352 1498 catalog_manager.cc:4841] TS bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195:44773): tablet e4181ac3118948a18e694eb5629227db (table test-table [id=a25719c4f2d04760809ad041f931c95b]) successfully deleted
I20260620 23:28:21.560667 2052 ts_tablet_manager.cc:1428] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:28:21.562350 2050 ts_tablet_manager.cc:1918] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:21.564415 2050 log.cc:1196] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/e4181ac3118948a18e694eb5629227db
I20260620 23:28:21.565086 2050 ts_tablet_manager.cc:1939] T e4181ac3118948a18e694eb5629227db P db4ac78a531e4fd998d5fb3dca6a784b: Deleting consensus metadata
I20260620 23:28:21.566987 1499 catalog_manager.cc:4841] TS db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975): tablet e4181ac3118948a18e694eb5629227db (table test-table [id=a25719c4f2d04760809ad041f931c95b]) successfully deleted
I20260620 23:28:21.566996 2049 ts_tablet_manager.cc:1918] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:21.567255 2049 log.cc:1196] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/e4181ac3118948a18e694eb5629227db
I20260620 23:28:21.568603 2049 ts_tablet_manager.cc:1939] T e4181ac3118948a18e694eb5629227db P c744552582d24932be55167ff0a0f199: Deleting consensus metadata
I20260620 23:28:21.570883 2051 tablet_bootstrap.cc:492] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: No bootstrap required, opened a new log
I20260620 23:28:21.571259 2051 ts_tablet_manager.cc:1397] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.004s
I20260620 23:28:21.571569 1501 catalog_manager.cc:4841] TS c744552582d24932be55167ff0a0f199 (127.0.96.193:46305): tablet e4181ac3118948a18e694eb5629227db (table test-table [id=a25719c4f2d04760809ad041f931c95b]) successfully deleted
I20260620 23:28:21.572508 2051 raft_consensus.cc:348] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.572803 2051 raft_consensus.cc:374] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:21.572947 2051 raft_consensus.cc:729] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc34ab1c25e84cadb13fabcf0524d3f6, State: Initialized, Role: FOLLOWER
I20260620 23:28:21.573294 2051 consensus_queue.cc:260] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.574095 2051 ts_tablet_manager.cc:1428] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:28:21.575982 2038 raft_consensus.cc:482] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:21.576238 2038 raft_consensus.cc:504] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.578361 2038 leader_election.cc:288] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975), bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195:44773)
I20260620 23:28:21.587293 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.578837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:32975 (local address 127.0.96.193:39773)
0620 23:28:21.579815 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.579839 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.580065 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.580688 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.580693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.580727 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:21.581167 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.581177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.582724 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.582730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.584419 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.584433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.585790 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.585797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.585985 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.586011 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.586128 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.586918 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":973,"threads_started":1}
I20260620 23:28:21.586675 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.579596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:39773 (local address 127.0.96.194:32975)
0620 23:28:21.579809 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.579815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.579931 (+ 116us) server_negotiation.cc:407] Connection header received
0620 23:28:21.580263 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.580271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.580375 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.580451 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:21.581334 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.582535 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.584605 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.585561 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.585756 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.585847 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.586183 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.586333 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:28:21.589004 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d34ca5b6547a6be822d986afc308e" candidate_uuid: "c744552582d24932be55167ff0a0f199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" is_pre_election: true
I20260620 23:28:21.589776 1816 raft_consensus.cc:2454] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c744552582d24932be55167ff0a0f199 in term 0.
I20260620 23:28:21.590161 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.580056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:53063 (local address 127.0.96.195:44773)
0620 23:28:21.580251 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.580257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.582015 (+ 1758us) server_negotiation.cc:407] Connection header received
0620 23:28:21.582182 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.582188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.582282 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.582425 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:21.583252 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.584474 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.587171 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.588472 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.588617 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.588695 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.589671 (+ 976us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.589848 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:28:21.590221 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.579149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:44773 (local address 127.0.96.193:53063)
0620 23:28:21.581864 (+ 2715us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.581886 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.582031 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.582451 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.582456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.582483 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:21.583033 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.583044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.584698 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.584706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.586974 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.586994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.589254 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.589259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.589471 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.589500 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.589619 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.589787 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2550,"mutex_wait_us":1,"thread_start_us":2299,"threads_started":1}
I20260620 23:28:21.591189 1602 leader_election.cc:302] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c744552582d24932be55167ff0a0f199, db4ac78a531e4fd998d5fb3dca6a784b; no voters:
I20260620 23:28:21.591741 2038 raft_consensus.cc:2781] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:21.591449 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d34ca5b6547a6be822d986afc308e" candidate_uuid: "c744552582d24932be55167ff0a0f199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" is_pre_election: true
I20260620 23:28:21.591889 2038 raft_consensus.cc:482] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:21.591990 2038 raft_consensus.cc:3037] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.592144 1955 raft_consensus.cc:2454] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c744552582d24932be55167ff0a0f199 in term 0.
I20260620 23:28:21.595057 2038 raft_consensus.cc:504] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.596026 2038 leader_election.cc:288] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [CANDIDATE]: Term 1 election: Requested vote from peers db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975), bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195:44773)
I20260620 23:28:21.596732 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d34ca5b6547a6be822d986afc308e" candidate_uuid: "c744552582d24932be55167ff0a0f199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4ac78a531e4fd998d5fb3dca6a784b"
I20260620 23:28:21.596837 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380d34ca5b6547a6be822d986afc308e" candidate_uuid: "c744552582d24932be55167ff0a0f199" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6"
I20260620 23:28:21.597051 1816 raft_consensus.cc:3037] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.597131 1955 raft_consensus.cc:3037] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.599367 1816 raft_consensus.cc:2454] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c744552582d24932be55167ff0a0f199 in term 1.
I20260620 23:28:21.599969 1602 leader_election.cc:302] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c744552582d24932be55167ff0a0f199, db4ac78a531e4fd998d5fb3dca6a784b; no voters:
I20260620 23:28:21.600387 2038 raft_consensus.cc:2781] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:21.601083 1955 raft_consensus.cc:2454] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c744552582d24932be55167ff0a0f199 in term 1.
I20260620 23:28:21.601707 2038 raft_consensus.cc:686] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 1 LEADER]: Becoming Leader. State: Replica: c744552582d24932be55167ff0a0f199, State: Running, Role: LEADER
I20260620 23:28:21.602229 2038 consensus_queue.cc:237] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.606348 1513 catalog_manager.cc:5495] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 reported cstate change: term changed from 0 to 1, leader changed from <none> to c744552582d24932be55167ff0a0f199 (127.0.96.193). New cstate: current_term: 1 leader_uuid: "c744552582d24932be55167ff0a0f199" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:28:21 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:21.671813 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.652225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42049 (local address 127.0.0.1:49544)
0620 23:28:21.654252 (+ 2027us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.654279 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.654490 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.654966 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.654969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.655799 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:21.656071 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.656081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.657705 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.657709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.658804 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.658815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.659869 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.659874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.660032 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.660885 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:21.660904 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:21.663484 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:21.668214 (+ 4730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:21.668233 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:21.668243 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:21.668834 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:21.669387 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:21.669391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:21.669435 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:21.669663 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:21.670491 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:21.670545 (+ 54us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:21.670762 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.670996 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.671350 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1867,"thread_start_us":1715,"threads_started":1}
I20260620 23:28:21.672113 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.652230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.0.96.254:42049)
0620 23:28:21.653056 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.653062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.654429 (+ 1367us) server_negotiation.cc:407] Connection header received
0620 23:28:21.654639 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.654646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.654749 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.654901 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:21.656295 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.657533 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.658957 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.659662 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.659808 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.663751 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:21.663759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:21.663764 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:21.663815 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:21.667933 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:21.669025 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:21.669031 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:21.669036 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:21.669217 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:21.669795 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:21.669800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:21.669804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:21.670254 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:21.670396 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.671110 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.671548 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":158,"server-negotiator.queue_time_us":610,"thread_start_us":399,"threads_started":1}
I20260620 23:28:21.679260 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:49544:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:28:21.680639 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:21.689422 1650 tablet_service.cc:1467] Processing CreateTablet for tablet fd2a651645904ca187966a2e88bbdd33 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f61d75268654779a0a76d5abd55e8df]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:28:21.689761 1791 tablet_service.cc:1467] Processing CreateTablet for tablet fd2a651645904ca187966a2e88bbdd33 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f61d75268654779a0a76d5abd55e8df]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:28:21.690024 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2a651645904ca187966a2e88bbdd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:21.690424 1791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2a651645904ca187966a2e88bbdd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:21.696576 2052 tablet_bootstrap.cc:492] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: Bootstrap starting.
I20260620 23:28:21.697719 2053 tablet_bootstrap.cc:492] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: Bootstrap starting.
I20260620 23:28:21.698629 1930 tablet_service.cc:1467] Processing CreateTablet for tablet fd2a651645904ca187966a2e88bbdd33 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3f61d75268654779a0a76d5abd55e8df]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:28:21.699348 1930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2a651645904ca187966a2e88bbdd33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:21.699520 2052 tablet_bootstrap.cc:654] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:21.699527 2053 tablet_bootstrap.cc:654] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:21.710646 2052 tablet_bootstrap.cc:492] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: No bootstrap required, opened a new log
I20260620 23:28:21.711063 2052 ts_tablet_manager.cc:1397] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260620 23:28:21.712074 2052 raft_consensus.cc:348] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.712436 2051 tablet_bootstrap.cc:492] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: Bootstrap starting.
I20260620 23:28:21.712496 2052 raft_consensus.cc:374] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:21.712674 2052 raft_consensus.cc:729] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db4ac78a531e4fd998d5fb3dca6a784b, State: Initialized, Role: FOLLOWER
I20260620 23:28:21.713124 2052 consensus_queue.cc:260] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.713773 2037 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.714190 2037 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.715052 2051 tablet_bootstrap.cc:654] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:21.719710 2052 ts_tablet_manager.cc:1428] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260620 23:28:21.721347 2053 tablet_bootstrap.cc:492] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: No bootstrap required, opened a new log
I20260620 23:28:21.722236 2053 ts_tablet_manager.cc:1397] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.007s
I20260620 23:28:21.723093 2053 raft_consensus.cc:348] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.723474 2053 raft_consensus.cc:374] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:21.723589 2053 raft_consensus.cc:729] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c744552582d24932be55167ff0a0f199, State: Initialized, Role: FOLLOWER
I20260620 23:28:21.723918 2053 consensus_queue.cc:260] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.724799 2038 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.725116 2038 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.726107 2053 ts_tablet_manager.cc:1428] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:28:21.726390 2051 tablet_bootstrap.cc:492] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: No bootstrap required, opened a new log
I20260620 23:28:21.726596 2051 ts_tablet_manager.cc:1397] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260620 23:28:21.728440 2051 raft_consensus.cc:348] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.728849 2051 raft_consensus.cc:374] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:21.728950 2051 raft_consensus.cc:729] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc34ab1c25e84cadb13fabcf0524d3f6, State: Initialized, Role: FOLLOWER
I20260620 23:28:21.729321 2051 consensus_queue.cc:260] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.729876 2027 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.730189 2027 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.730944 2051 ts_tablet_manager.cc:1428] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:21.746671 2027 raft_consensus.cc:482] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:21.746908 2027 raft_consensus.cc:504] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.747716 2027 leader_election.cc:288] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c744552582d24932be55167ff0a0f199 (127.0.96.193:46305), db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975)
I20260620 23:28:21.748553 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2a651645904ca187966a2e88bbdd33" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" is_pre_election: true
I20260620 23:28:21.748567 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2a651645904ca187966a2e88bbdd33" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c744552582d24932be55167ff0a0f199" is_pre_election: true
I20260620 23:28:21.748858 1816 raft_consensus.cc:2454] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 0.
I20260620 23:28:21.748904 1671 raft_consensus.cc:2454] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 0.
I20260620 23:28:21.749389 1882 leader_election.cc:302] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc34ab1c25e84cadb13fabcf0524d3f6, db4ac78a531e4fd998d5fb3dca6a784b; no voters:
I20260620 23:28:21.749716 2027 raft_consensus.cc:2781] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:21.749842 2027 raft_consensus.cc:482] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:21.749944 2027 raft_consensus.cc:3037] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.753897 2027 raft_consensus.cc:504] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.754824 2027 leader_election.cc:288] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 election: Requested vote from peers c744552582d24932be55167ff0a0f199 (127.0.96.193:46305), db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975)
I20260620 23:28:21.755577 1816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2a651645904ca187966a2e88bbdd33" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db4ac78a531e4fd998d5fb3dca6a784b"
I20260620 23:28:21.755861 1816 raft_consensus.cc:3037] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.758220 1671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2a651645904ca187966a2e88bbdd33" candidate_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c744552582d24932be55167ff0a0f199"
I20260620 23:28:21.758514 1671 raft_consensus.cc:3037] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:21.759068 1816 raft_consensus.cc:2454] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 1.
I20260620 23:28:21.759604 1882 leader_election.cc:302] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc34ab1c25e84cadb13fabcf0524d3f6, db4ac78a531e4fd998d5fb3dca6a784b; no voters:
I20260620 23:28:21.760002 2027 raft_consensus.cc:2781] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:21.761013 2027 raft_consensus.cc:686] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 LEADER]: Becoming Leader. State: Replica: bc34ab1c25e84cadb13fabcf0524d3f6, State: Running, Role: LEADER
I20260620 23:28:21.761317 2027 consensus_queue.cc:237] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } }
I20260620 23:28:21.761698 1671 raft_consensus.cc:2454] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc34ab1c25e84cadb13fabcf0524d3f6 in term 1.
I20260620 23:28:21.763381 2031 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: TxnStatusTablet state changed. Reason: New leader bc34ab1c25e84cadb13fabcf0524d3f6. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.763723 2031 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:28:21.764760 2072 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:28:21.765098 2072 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:28:21.765892 1513 catalog_manager.cc:5495] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } health_report { overall_health: HEALTHY } } }
I20260620 23:28:21.794476 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.787564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35716 (local address 127.0.96.195:44773)
0620 23:28:21.787847 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.787855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.788064 (+ 209us) server_negotiation.cc:407] Connection header received
0620 23:28:21.788318 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.788323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.788408 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.788525 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:21.789062 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.790000 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.792471 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.792973 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.793158 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.793723 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.793854 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.793930 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.794117 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.794207 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260620 23:28:21.794580 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.787563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:44773 (local address 127.0.0.1:35716)
0620 23:28:21.787881 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.787907 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.788098 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.788579 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.788584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.788597 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:21.788809 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.788820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.790165 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.790171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.792267 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.792285 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.793222 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.793229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.793416 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.793498 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.793963 (+ 465us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.793989 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.794082 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.794262 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 23:28:21.799191 1816 raft_consensus.cc:1261] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Refusing update from remote peer bc34ab1c25e84cadb13fabcf0524d3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:21.799152 1671 raft_consensus.cc:1261] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Refusing update from remote peer bc34ab1c25e84cadb13fabcf0524d3f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:21.799963 2027 consensus_queue.cc:1035] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:21.800362 2031 consensus_queue.cc:1035] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:21.804023 2037 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: TxnStatusTablet state changed. Reason: New leader bc34ab1c25e84cadb13fabcf0524d3f6. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.804400 2037 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.808579 2045 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.808957 2045 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:28:21.810793 2039 mvcc.cc:205] Tried to move back new op lower bound from 7299064224961855488 to 7299064224817455104. Current Snapshot: MvccSnapshot[applied={T|T < 7299064224961855488}]
I20260620 23:28:21.813158 2037 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.813398 2037 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P db4ac78a531e4fd998d5fb3dca6a784b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.818743 2031 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.819011 2031 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P bc34ab1c25e84cadb13fabcf0524d3f6: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:28:21.819103 2038 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: TxnStatusTablet state changed. Reason: New leader bc34ab1c25e84cadb13fabcf0524d3f6. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.819463 2038 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.822676 2038 tablet_replica.cc:439] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c744552582d24932be55167ff0a0f199" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 46305 } } peers { permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 } } peers { permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 } } }
I20260620 23:28:21.823041 2038 tablet_replica.cc:442] T fd2a651645904ca187966a2e88bbdd33 P c744552582d24932be55167ff0a0f199: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:21.833559 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.822846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.96.193:46305)
0620 23:28:21.823574 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.823580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.823604 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:28:21.823721 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.823726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.823837 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.824612 (+ 775us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:21.824883 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.827008 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.829235 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.829972 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.830726 (+ 754us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.830933 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.832002 (+ 1069us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.832910 (+ 908us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.833027 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.833138 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":1839,"threads_started":1}
I20260620 23:28:21.856314 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.848873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:40839 (local address 127.0.96.195:44773)
0620 23:28:21.849101 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.849106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.849896 (+ 790us) server_negotiation.cc:407] Connection header received
0620 23:28:21.850096 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.850103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.850219 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.850366 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:21.850968 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.852107 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.853900 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.854835 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.854986 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.855490 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.855630 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.855748 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.855884 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.855965 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:28:21.856400 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.848906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:44773 (local address 127.0.96.193:40839)
0620 23:28:21.849745 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.849764 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.849917 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.850427 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.850432 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.850443 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:21.850766 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.850775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.852308 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.852313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.853693 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.853707 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.855046 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.855053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.855215 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.855274 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.855747 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.855760 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.855862 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.855996 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":482,"threads_started":1}
I20260620 23:28:21.867386 1816 raft_consensus.cc:1261] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Refusing update from remote peer c744552582d24932be55167ff0a0f199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:21.867699 1955 raft_consensus.cc:1261] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Refusing update from remote peer c744552582d24932be55167ff0a0f199: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:21.868181 2059 consensus_queue.cc:1035] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db4ac78a531e4fd998d5fb3dca6a784b" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 32975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:21.868584 2038 consensus_queue.cc:1035] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc34ab1c25e84cadb13fabcf0524d3f6" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 44773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:21.918650 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.910662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:36041 (local address 127.0.96.193:46305)
0620 23:28:21.910905 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:21.910913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:21.911583 (+ 670us) server_negotiation.cc:407] Connection header received
0620 23:28:21.911799 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:21.911804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:21.911883 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:21.911978 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:21.912569 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.913499 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.916448 (+ 2949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.916862 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.916972 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.917732 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.917892 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.917992 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:21.918202 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:28:21.918350 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:28:21.919173 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:21.910673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:46305 (local address 127.0.96.195:36041)
0620 23:28:21.911464 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:21.911481 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:21.911659 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:21.911993 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:21.911997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:21.912009 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:21.912406 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.912415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.913689 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.913696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.916243 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:21.916270 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:21.917171 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:21.917179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:21.917399 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:21.917512 (+ 113us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:21.918030 (+ 518us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:21.918044 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:28:21.918152 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:28:21.918346 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":429,"threads_started":1}
I20260620 23:28:21.965410 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:21.965824 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:21.970491 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ed5faeb2f904149beac974967d40fb5: Sending DeleteTablet for 3 replicas of tablet 380d34ca5b6547a6be822d986afc308e
I20260620 23:28:21.971681 1791 tablet_service.cc:1514] Processing DeleteTablet for tablet 380d34ca5b6547a6be822d986afc308e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:21 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:46298
I20260620 23:28:21.972033 2050 tablet_replica.cc:330] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: stopping tablet replica
I20260620 23:28:21.972239 2050 raft_consensus.cc:2229] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:21.972036 1930 tablet_service.cc:1514] Processing DeleteTablet for tablet 380d34ca5b6547a6be822d986afc308e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:21 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:35698
I20260620 23:28:21.973085 2050 raft_consensus.cc:2258] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:21.973547 1650 tablet_service.cc:1514] Processing DeleteTablet for tablet 380d34ca5b6547a6be822d986afc308e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:21 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:59724
I20260620 23:28:21.973922 2049 tablet_replica.cc:330] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: stopping tablet replica
I20260620 23:28:21.974188 2049 raft_consensus.cc:2229] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:21.974817 2050 ts_tablet_manager.cc:1905] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:21.974848 2048 tablet_replica.cc:330] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: stopping tablet replica
I20260620 23:28:21.974891 2049 raft_consensus.cc:2258] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:21.975085 2048 raft_consensus.cc:2229] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:21.975319 2048 raft_consensus.cc:2258] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:21.976706 2049 ts_tablet_manager.cc:1905] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:21.978346 2048 ts_tablet_manager.cc:1905] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:21.988201 2049 ts_tablet_manager.cc:1918] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:21.988468 2049 log.cc:1196] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/380d34ca5b6547a6be822d986afc308e
I20260620 23:28:21.989053 2049 ts_tablet_manager.cc:1939] T 380d34ca5b6547a6be822d986afc308e P c744552582d24932be55167ff0a0f199: Deleting consensus metadata
I20260620 23:28:21.991005 1501 catalog_manager.cc:4841] TS c744552582d24932be55167ff0a0f199 (127.0.96.193:46305): tablet 380d34ca5b6547a6be822d986afc308e (table test-table [id=d635074e1f3b436f8df6f2fd842c87ed]) successfully deleted
I20260620 23:28:21.993415 2048 ts_tablet_manager.cc:1918] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:21.993664 2048 log.cc:1196] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/380d34ca5b6547a6be822d986afc308e
I20260620 23:28:21.994210 2048 ts_tablet_manager.cc:1939] T 380d34ca5b6547a6be822d986afc308e P bc34ab1c25e84cadb13fabcf0524d3f6: Deleting consensus metadata
I20260620 23:28:21.995015 2050 ts_tablet_manager.cc:1918] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:21.995275 2050 log.cc:1196] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/380d34ca5b6547a6be822d986afc308e
I20260620 23:28:21.995844 2050 ts_tablet_manager.cc:1939] T 380d34ca5b6547a6be822d986afc308e P db4ac78a531e4fd998d5fb3dca6a784b: Deleting consensus metadata
I20260620 23:28:21.995811 1498 catalog_manager.cc:4841] TS bc34ab1c25e84cadb13fabcf0524d3f6 (127.0.96.195:44773): tablet 380d34ca5b6547a6be822d986afc308e (table test-table [id=d635074e1f3b436f8df6f2fd842c87ed]) successfully deleted
I20260620 23:28:21.997164 1499 catalog_manager.cc:4841] TS db4ac78a531e4fd998d5fb3dca6a784b (127.0.96.194:32975): tablet 380d34ca5b6547a6be822d986afc308e (table test-table [id=d635074e1f3b436f8df6f2fd842c87ed]) successfully deleted
Jun 20 23:28:22 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:22.032935 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:22.002319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59742 (local address 127.0.96.193:46305)
0620 23:28:22.002568 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:22.002575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:22.005972 (+ 3397us) server_negotiation.cc:407] Connection header received
0620 23:28:22.006222 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:22.006228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:22.006357 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:22.006528 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:22.009110 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.010361 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.012559 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.013464 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.013649 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:22.022724 (+ 9075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:22.022769 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:22.022792 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:22.022872 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:22.028572 (+ 5700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.029578 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.029588 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.029591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.029743 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.030224 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.030230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.030233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.030719 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:22.031022 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:22.032300 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0620 23:28:22.032515 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 23:28:22.036118 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59742
I20260620 23:28:22.076835 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:22.042751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.0.96.254:42049)
0620 23:28:22.043057 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:22.043064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:22.053824 (+ 10760us) server_negotiation.cc:407] Connection header received
0620 23:28:22.057524 (+ 3700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:22.057532 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:22.057665 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:22.057766 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:22.060228 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.061492 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.063502 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.064345 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.064509 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:22.068154 (+ 3645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:22.068162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:22.068167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:22.068218 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:22.072390 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.073595 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.073600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.073617 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.073769 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.074363 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.074369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.074372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.074731 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:22.074976 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:22.076254 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0620 23:28:22.076487 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 23:28:22.078696 1514 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49556
I20260620 23:28:22.081729 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1585
I20260620 23:28:22.104070 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:22.085317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.96.193:46305)
0620 23:28:22.085510 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:22.085516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:22.085535 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:28:22.085592 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:22.085597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:22.085691 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:22.085789 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:22.087674 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.089286 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.091191 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.091977 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.092211 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:22.095396 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:22.095404 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:22.095414 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:22.095475 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:22.099693 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.101154 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.101162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.101166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.101376 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.101901 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.101906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.101910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.102392 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:22.102680 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:22.103035 (+ 355us) server_negotiation.cc:299] Negotiation successful
0620 23:28:22.103319 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 23:28:22.104862 1715 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59752
W20260620 23:28:22.105651 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:22.105876 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1585
I20260620 23:28:22.126447 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1725
Jun 20 23:28:22 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:22.146613 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:22.127317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46302 (local address 127.0.96.194:32975)
0620 23:28:22.127857 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:22.127861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:22.127881 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:28:22.127941 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:22.127945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:22.128010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:22.128113 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:22.129882 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.130690 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.132560 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.132948 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.133220 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:22.138785 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:22.138799 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:22.138815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:22.138855 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:22.142985 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.144213 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.144223 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.144227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.144403 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.144879 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.144885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.144888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.145311 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:22.145600 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:22.145996 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 23:28:22.146234 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":306,"threads_started":1}
W20260620 23:28:22.147392 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46302
W20260620 23:28:22.148231 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:22.148449 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1725
I20260620 23:28:22.168686 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1865
Jun 20 23:28:22 dist-test-slave-v34r krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:22.191601 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:22.169851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.0.96.195:44773)
0620 23:28:22.170120 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:22.170127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:22.170155 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:28:22.170276 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:22.170282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:22.170377 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:22.170506 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:22.172409 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.173627 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.175136 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:22.175871 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:22.176014 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:22.182003 (+ 5989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:22.182026 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:22.182053 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:22.182123 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:22.187454 (+ 5331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.188657 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.188669 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.188673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.188848 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:22.189442 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:22.189447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:22.189451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:22.190105 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:22.190496 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:22.190843 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 23:28:22.191139 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 23:28:22.192538 1996 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35730
W20260620 23:28:22.193463 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1865: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:22.193718 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1865
I20260620 23:28:22.212459 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 1482
I20260620 23:28:22.213827 1554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:22.330734 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 1482
2026-06-20T23:28:22Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5310 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:22 dist-test-slave-v34r krb5kdc[2098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:22 dist-test-slave-v34r krb5kdc[2098](info): set up 2 sockets
Jun 20 23:28:22 dist-test-slave-v34r krb5kdc[2098](info): commencing operation
krb5kdc: starting...
W20260620 23:28:23.433413 387 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:23 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:23.647222 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38901
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:23.985603 2111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:23.985980 2111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:23.986119 2111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:23.993498 2111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:23.993618 2111 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:28:23.993660 2111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:23.993716 2111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:23.993760 2111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:23.993829 2111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:24.004109 2111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38901
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:24.005483 2111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:24.006892 2111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:24.020771 2116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:24.021039 2117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:24.020431 2119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:24.022603 2111 server_base.cc:1034] running on GCE node
I20260620 23:28:24.023806 2111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:24.025938 2111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:24.029326 2111 hybrid_clock.cc:648] HybridClock initialized: now 1781998104029233 us; error 79 us; skew 500 ppm
I20260620 23:28:24.029697 2111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:24 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:24.033447 2111 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:24.035487 2111 webserver.cc:460] Webserver started at http://127.0.96.254:39435/ using document root <none> and password file <none>
I20260620 23:28:24.036402 2111 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:24.036527 2111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:24.036872 2111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:24.040726 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "98f354658d324f6ab4830c0cbb9f68e0"
format_stamp: "Formatted at 2026-06-20 23:28:24 on dist-test-slave-v34r"
server_key: "cd24ffe0af7e1c1232ea1fce80cd326c"
server_key_iv: "c01a682bdf518ff1920b53f5ceb5f24c"
server_key_version: "encryptionkey@0"
I20260620 23:28:24.041774 2111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "98f354658d324f6ab4830c0cbb9f68e0"
format_stamp: "Formatted at 2026-06-20 23:28:24 on dist-test-slave-v34r"
server_key: "cd24ffe0af7e1c1232ea1fce80cd326c"
server_key_iv: "c01a682bdf518ff1920b53f5ceb5f24c"
server_key_version: "encryptionkey@0"
I20260620 23:28:24.048924 2111 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:28:24.053825 2126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:24.055776 2111 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:24.056061 2111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "98f354658d324f6ab4830c0cbb9f68e0"
format_stamp: "Formatted at 2026-06-20 23:28:24 on dist-test-slave-v34r"
server_key: "cd24ffe0af7e1c1232ea1fce80cd326c"
server_key_iv: "c01a682bdf518ff1920b53f5ceb5f24c"
server_key_version: "encryptionkey@0"
I20260620 23:28:24.056388 2111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:24.072320 2111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:24.078276 2111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:24.078595 2111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:24.103214 2111 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38901
I20260620 23:28:24.103250 2187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38901 every 8 connection(s)
I20260620 23:28:24.104745 2111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:24.109282 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 2111
I20260620 23:28:24.109539 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:24.110189 387 external_mini_cluster.cc:1428] Setting key e70ed5ca8554363818c035e4aae71846
I20260620 23:28:24.110657 2188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:28:24 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:24.128841 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Bootstrap starting.
I20260620 23:28:24.134054 2188 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:24.135466 2188 log.cc:824] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:24.139457 2191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:24.112702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54860 (local address 127.0.96.254:38901)
0620 23:28:24.113751 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:24.113773 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:24.113843 (+ 70us) server_negotiation.cc:407] Connection header received
0620 23:28:24.116858 (+ 3015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:24.116907 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:24.117732 (+ 825us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:24.118389 (+ 657us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:24.119970 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.122173 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.123620 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.125192 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.125515 (+ 323us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:24.130168 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:24.130182 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:24.130209 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:24.130278 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:24.134798 (+ 4520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:24.135618 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:24.135625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:24.135633 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:24.135873 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:24.136264 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:24.136269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:24.136272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:24.136963 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:24.137365 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:24.137823 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 23:28:24.138296 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":448,"threads_started":1}
I20260620 23:28:24.139757 2188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: No bootstrap required, opened a new log
I20260620 23:28:24.149848 2188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f354658d324f6ab4830c0cbb9f68e0" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38901 } }
I20260620 23:28:24.150410 2188 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:24.150534 2188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f354658d324f6ab4830c0cbb9f68e0, State: Initialized, Role: FOLLOWER
I20260620 23:28:24.151307 2188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f354658d324f6ab4830c0cbb9f68e0" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38901 } }
I20260620 23:28:24.151618 2188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:24.151779 2188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:24.151973 2188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:24.155383 2188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f354658d324f6ab4830c0cbb9f68e0" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38901 } }
I20260620 23:28:24.156383 2188 leader_election.cc:302] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f354658d324f6ab4830c0cbb9f68e0; no voters:
I20260620 23:28:24.157207 2188 leader_election.cc:288] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:24.157410 2193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:24.158394 2193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [term 1 LEADER]: Becoming Leader. State: Replica: 98f354658d324f6ab4830c0cbb9f68e0, State: Running, Role: LEADER
I20260620 23:28:24.159087 2193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f354658d324f6ab4830c0cbb9f68e0" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38901 } }
I20260620 23:28:24.160183 2188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:24.162691 2195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f354658d324f6ab4830c0cbb9f68e0. Latest consensus state: current_term: 1 leader_uuid: "98f354658d324f6ab4830c0cbb9f68e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f354658d324f6ab4830c0cbb9f68e0" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38901 } } }
I20260620 23:28:24.163115 2195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:24.163659 2194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f354658d324f6ab4830c0cbb9f68e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f354658d324f6ab4830c0cbb9f68e0" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38901 } } }
I20260620 23:28:24.164116 2194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:24.165169 2198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:24.171468 2198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:24.181766 2198 catalog_manager.cc:1347] Generated new cluster ID: 229847cdee5a4933bd263fbccd5a24fd
I20260620 23:28:24.181926 2198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:24.195338 2198 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:24.196835 2198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:24.209841 2198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Generated new TSK 0
I20260620 23:28:24.210764 2198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:24.266899 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38901
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:24.636847 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:24.637368 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:24.637552 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:24.646095 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:24.646255 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:24.646476 2214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:28:24.646564 2214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:28:24.646713 2214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:24.658442 2214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:24.660264 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:24.662274 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:24.678737 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:24.679227 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:24.679440 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:24.682276 2214 server_base.cc:1034] running on GCE node
I20260620 23:28:24.683140 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:24.684448 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:24.685699 2214 hybrid_clock.cc:648] HybridClock initialized: now 1781998104685630 us; error 61 us; skew 500 ppm
I20260620 23:28:24.686149 2214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:24 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:24.696197 2214 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:24.698473 2214 webserver.cc:460] Webserver started at http://127.0.96.193:41621/ using document root <none> and password file <none>
I20260620 23:28:24.699508 2214 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:24.699630 2214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:24.699913 2214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:24.703403 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "4186cebe1f2e45a08fae117df36fbc02"
format_stamp: "Formatted at 2026-06-20 23:28:24 on dist-test-slave-v34r"
server_key: "00b01082a25e84d55776efdaea38d197"
server_key_iv: "e2aa2e729dae8e7a2587962881189b22"
server_key_version: "encryptionkey@0"
I20260620 23:28:24.704254 2214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "4186cebe1f2e45a08fae117df36fbc02"
format_stamp: "Formatted at 2026-06-20 23:28:24 on dist-test-slave-v34r"
server_key: "00b01082a25e84d55776efdaea38d197"
server_key_iv: "e2aa2e729dae8e7a2587962881189b22"
server_key_version: "encryptionkey@0"
I20260620 23:28:24.710668 2214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 23:28:24.715279 2229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:24.717360 2214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:28:24.717600 2214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "4186cebe1f2e45a08fae117df36fbc02"
format_stamp: "Formatted at 2026-06-20 23:28:24 on dist-test-slave-v34r"
server_key: "00b01082a25e84d55776efdaea38d197"
server_key_iv: "e2aa2e729dae8e7a2587962881189b22"
server_key_version: "encryptionkey@0"
I20260620 23:28:24.717819 2214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:24.774974 2214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:24.779874 2214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:24.780198 2214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:24.783481 2214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:24.783583 2214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:24.783725 2214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:24.783805 2214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:28:24 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:24.817194 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:24.784698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:42509 (local address 127.0.96.254:38901)
0620 23:28:24.785381 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:24.785389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:24.787079 (+ 1690us) server_negotiation.cc:407] Connection header received
0620 23:28:24.790654 (+ 3575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:24.790662 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:24.790794 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:24.790940 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:24.793949 (+ 3009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.795296 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.798274 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.799079 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.799279 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:24.805460 (+ 6181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:24.805469 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:24.805478 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:24.805541 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:24.809230 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:24.810783 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:24.810794 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:24.810801 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:24.811075 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:24.811909 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:24.811918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:24.811923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:24.812372 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:24.812556 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:24.816573 (+ 4017us) server_negotiation.cc:299] Negotiation successful
0620 23:28:24.816833 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":344,"threads_started":1}
I20260620 23:28:24.817564 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:24.785665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.96.193:42509)
0620 23:28:24.786813 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:24.786874 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:24.790121 (+ 3247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:24.791337 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:24.791361 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:24.792569 (+ 1208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:24.793593 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:24.793638 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.795600 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.795612 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:24.798013 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:24.798047 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.799444 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.799456 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:24.799806 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:24.800902 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:24.800937 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:24.804820 (+ 3883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:24.809445 (+ 4625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:24.809462 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:24.809499 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:24.810493 (+ 994us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:24.811255 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:24.811260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:24.811262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:24.811701 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:24.812714 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:24.812730 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:24.813810 (+ 1080us) client_negotiation.cc:769] Sending connection context
0620 23:28:24.814371 (+ 561us) client_negotiation.cc:240] Negotiation successful
0620 23:28:24.815345 (+ 974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":515,"threads_started":1}
I20260620 23:28:24.840919 2214 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:43395
I20260620 23:28:24.840969 2349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:43395 every 8 connection(s)
I20260620 23:28:24.842722 2214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:24.845685 2350 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38901
I20260620 23:28:24.845968 2350 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:24.846787 2238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:24.828266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:49877 (local address 127.0.96.254:38901)
0620 23:28:24.828476 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:24.828485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:24.829639 (+ 1154us) server_negotiation.cc:407] Connection header received
0620 23:28:24.829903 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:24.829909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:24.829985 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:24.830091 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:24.831077 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.832241 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.835535 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.836341 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.836492 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:24.839617 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:24.839624 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:24.839628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:24.839667 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:24.843021 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:24.844075 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:24.844087 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:24.844094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:24.844368 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:24.844938 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:24.844943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:24.844945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:24.845435 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:24.845591 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:24.846175 (+ 584us) server_negotiation.cc:299] Negotiation successful
0620 23:28:24.846416 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:28:24.846822 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:24.828507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.96.193:49877)
0620 23:28:24.829504 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:24.829530 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:24.829738 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:24.830128 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:24.830133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:24.830648 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:24.830931 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:24.830940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.832465 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.832477 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:24.835283 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:24.835305 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:24.836817 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:24.836832 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:24.837308 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:24.838361 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:24.838401 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:24.839334 (+ 933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:24.843163 (+ 3829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:24.843169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:24.843173 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:24.843825 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:24.844536 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:24.844541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:24.844545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:24.844738 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:24.845581 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:24.845588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:24.845855 (+ 267us) client_negotiation.cc:769] Sending connection context
0620 23:28:24.846001 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 23:28:24.846403 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":646,"threads_started":1}
I20260620 23:28:24.847015 2350 heartbeater.cc:511] Master 127.0.96.254:38901 requested a full tablet report, sending...
I20260620 23:28:24.848886 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 2214
I20260620 23:28:24.849171 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:24.849520 2142 ts_manager.cc:194] Registered new tserver with Master: 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395)
I20260620 23:28:24.849964 387 external_mini_cluster.cc:1428] Setting key 2a9a3aa88874aeff7d5cc5f0c012fbbd
I20260620 23:28:24.851751 2142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:42509
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
W20260620 23:28:24.987954 387 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.96.194: real 0.135s user 0.002s sys 0.008s
I20260620 23:28:24.988803 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38901
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:25.332677 2354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:25.333081 2354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:25.333266 2354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:25.340819 2354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:25.340983 2354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:25.341179 2354 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:28:25.341274 2354 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:28:25.341396 2354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:25.351754 2354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:25.353322 2354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:25.354717 2354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:25.370188 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:25.370236 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:25.370337 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:25.376608 2354 server_base.cc:1034] running on GCE node
I20260620 23:28:25.377628 2354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:25.378863 2354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:25.380142 2354 hybrid_clock.cc:648] HybridClock initialized: now 1781998105380050 us; error 65 us; skew 500 ppm
I20260620 23:28:25.380609 2354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:25 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:25.384719 2354 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:25.386809 2354 webserver.cc:460] Webserver started at http://127.0.96.194:46409/ using document root <none> and password file <none>
I20260620 23:28:25.387753 2354 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:25.387917 2354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:25.388298 2354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:25.392297 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "df90a9c4b5714ae4a6d4dc1ec232749c"
format_stamp: "Formatted at 2026-06-20 23:28:25 on dist-test-slave-v34r"
server_key: "4a2f994a4bc21494c8b6061bccf856ea"
server_key_iv: "073eedd6f5a5eb7e09dda4ca8a5b28fb"
server_key_version: "encryptionkey@0"
I20260620 23:28:25.393401 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "df90a9c4b5714ae4a6d4dc1ec232749c"
format_stamp: "Formatted at 2026-06-20 23:28:25 on dist-test-slave-v34r"
server_key: "4a2f994a4bc21494c8b6061bccf856ea"
server_key_iv: "073eedd6f5a5eb7e09dda4ca8a5b28fb"
server_key_version: "encryptionkey@0"
I20260620 23:28:25.400542 2354 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260620 23:28:25.406257 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:25.408428 2354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:25.408735 2354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "df90a9c4b5714ae4a6d4dc1ec232749c"
format_stamp: "Formatted at 2026-06-20 23:28:25 on dist-test-slave-v34r"
server_key: "4a2f994a4bc21494c8b6061bccf856ea"
server_key_iv: "073eedd6f5a5eb7e09dda4ca8a5b28fb"
server_key_version: "encryptionkey@0"
I20260620 23:28:25.409070 2354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:25.427718 2354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:25.434224 2354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:25.434640 2354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:25.438710 2354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:25.438867 2354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:25.439024 2354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:25.439131 2354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:28:25 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:25.470623 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:25.439589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:60315 (local address 127.0.96.254:38901)
0620 23:28:25.440221 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:25.440228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:25.442111 (+ 1883us) server_negotiation.cc:407] Connection header received
0620 23:28:25.445102 (+ 2991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:25.445110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:25.445226 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:25.445422 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:25.448660 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.450005 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.453840 (+ 3835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.454529 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.454739 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:25.461705 (+ 6966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:25.461712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:25.461715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:25.461758 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:25.464855 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:25.466607 (+ 1752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:25.466616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:25.466622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:25.466842 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:25.467431 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:25.467436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:25.467439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:25.467768 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:25.467951 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:25.469500 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0620 23:28:25.469903 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
I20260620 23:28:25.472872 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:25.440479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.96.194:60315)
0620 23:28:25.441772 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:25.441884 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:25.444583 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:25.445813 (+ 1230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:25.445829 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:25.447015 (+ 1186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:25.448355 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:25.448403 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.450330 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.450340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:25.453501 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:25.453548 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.454914 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.454928 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:25.455615 (+ 687us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:25.456874 (+ 1259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:25.456946 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:25.461286 (+ 4340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:25.465105 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:25.465119 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:25.465151 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:25.466339 (+ 1188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:25.466982 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:25.466987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:25.466995 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:25.467298 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:25.468073 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:25.468090 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:25.468832 (+ 742us) client_negotiation.cc:769] Sending connection context
0620 23:28:25.469330 (+ 498us) client_negotiation.cc:240] Negotiation successful
0620 23:28:25.470157 (+ 827us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":653,"threads_started":1}
I20260620 23:28:25.502835 2354 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34037
I20260620 23:28:25.502904 2489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34037 every 8 connection(s)
I20260620 23:28:25.504385 2354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:25.505934 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:25.481492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:52765 (local address 127.0.96.254:38901)
0620 23:28:25.481789 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:25.481799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:25.486373 (+ 4574us) server_negotiation.cc:407] Connection header received
0620 23:28:25.486720 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:25.486727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:25.486855 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:25.486979 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:25.489475 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.490428 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.493378 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.494221 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.494355 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:25.498315 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:25.498322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:25.498326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:25.498378 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:25.502115 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:25.503350 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:25.503355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:25.503358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:25.503519 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:25.503953 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:25.503956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:25.503958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:25.504345 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:25.504503 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:25.505147 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 23:28:25.505466 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260620 23:28:25.506340 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:25.481456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.96.194:52765)
0620 23:28:25.486165 (+ 4709us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:25.486205 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:25.486472 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:25.487081 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:25.487089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:25.488299 (+ 1210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:25.489137 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:25.489158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.490669 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.490681 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:25.493083 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:25.493105 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:25.494932 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:25.494944 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:25.495392 (+ 448us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:25.496335 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:25.496371 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:25.497980 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:25.502367 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:25.502375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:25.502379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:25.503100 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:25.503648 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:25.503652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:25.503655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:25.503809 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:25.504481 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:25.504486 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:25.504691 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 23:28:25.504854 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 23:28:25.505441 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4431,"thread_start_us":4169,"threads_started":1}
I20260620 23:28:25.509105 2490 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38901
I20260620 23:28:25.509336 2490 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:25.510200 2490 heartbeater.cc:511] Master 127.0.96.254:38901 requested a full tablet report, sending...
I20260620 23:28:25.511982 2143 ts_manager.cc:194] Registered new tserver with Master: df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037)
I20260620 23:28:25.512949 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 2354
I20260620 23:28:25.513199 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:25.513202 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:60315
I20260620 23:28:25.513680 387 external_mini_cluster.cc:1428] Setting key 6005b36061e83ebee29c2c31e6d27cc0
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:25.557073 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38901
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:28:25.855020 2350 heartbeater.cc:503] Master 127.0.96.254:38901 was elected leader, sending a full tablet report...
W20260620 23:28:25.897383 2494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:25.897734 2494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:25.897886 2494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:25.906098 2494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:25.906258 2494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:25.906508 2494 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 23:28:25.906631 2494 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:28:25.906821 2494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:25.918900 2494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:38901
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:25.920368 2494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:25.921954 2494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:25.938246 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:25.938232 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:25.938195 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:25.938938 2494 server_base.cc:1034] running on GCE node
I20260620 23:28:25.939904 2494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:25.941251 2494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:25.942519 2494 hybrid_clock.cc:648] HybridClock initialized: now 1781998105942442 us; error 81 us; skew 500 ppm
I20260620 23:28:25.942951 2494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:25 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:25.946888 2494 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:25.949210 2494 webserver.cc:460] Webserver started at http://127.0.96.195:38271/ using document root <none> and password file <none>
I20260620 23:28:25.950177 2494 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:25.950313 2494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:25.950651 2494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:25.954468 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "e4c9c36610194b32810b5eb1724258fd"
format_stamp: "Formatted at 2026-06-20 23:28:25 on dist-test-slave-v34r"
server_key: "1986f8b36e4c2ea00e1fd83f7effe571"
server_key_iv: "efb0a69ca99fb6cd8d9b4624b339b1f0"
server_key_version: "encryptionkey@0"
I20260620 23:28:25.955454 2494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "e4c9c36610194b32810b5eb1724258fd"
format_stamp: "Formatted at 2026-06-20 23:28:25 on dist-test-slave-v34r"
server_key: "1986f8b36e4c2ea00e1fd83f7effe571"
server_key_iv: "efb0a69ca99fb6cd8d9b4624b339b1f0"
server_key_version: "encryptionkey@0"
I20260620 23:28:25.963119 2494 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260620 23:28:25.968137 2509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:25.970131 2494 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:28:25.970384 2494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "e4c9c36610194b32810b5eb1724258fd"
format_stamp: "Formatted at 2026-06-20 23:28:25 on dist-test-slave-v34r"
server_key: "1986f8b36e4c2ea00e1fd83f7effe571"
server_key_iv: "efb0a69ca99fb6cd8d9b4624b339b1f0"
server_key_version: "encryptionkey@0"
I20260620 23:28:25.970691 2494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:25.998546 2494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:26.004092 2494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:26.004484 2494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:26.008147 2494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:26.008293 2494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:26.008493 2494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:26.008634 2494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:28:26 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:26.037205 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.009059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:47481 (local address 127.0.96.254:38901)
0620 23:28:26.009793 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.009799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.011342 (+ 1543us) server_negotiation.cc:407] Connection header received
0620 23:28:26.013955 (+ 2613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.013961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.014065 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.014196 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:26.016854 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.018063 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.021340 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.022072 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.022196 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.027966 (+ 5770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:26.027976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:26.027984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:26.028078 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:26.033091 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.034427 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.034434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.034440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.034583 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.035071 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.035078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.035084 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.035521 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:26.035667 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.036528 (+ 861us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.036748 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":503,"threads_started":1}
I20260620 23:28:26.038453 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.009987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.96.195:47481)
0620 23:28:26.011123 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.011178 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.013597 (+ 2419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.014500 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.014514 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.015473 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:26.016620 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.016661 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.018246 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.018254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.021103 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.021129 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.022367 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.022377 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.022759 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.023683 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:26.023708 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:26.027594 (+ 3886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:26.033346 (+ 5752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:26.033363 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:26.033387 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:26.034212 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:26.034711 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:26.034716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:26.034720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:26.034937 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:26.035650 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:26.035658 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:26.036116 (+ 458us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.036425 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.037086 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":532,"threads_started":1}
I20260620 23:28:26.067800 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.047002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:33933 (local address 127.0.96.254:38901)
0620 23:28:26.047349 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.047361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.050288 (+ 2927us) server_negotiation.cc:407] Connection header received
0620 23:28:26.050599 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.050608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.050719 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.050923 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:26.052049 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.052836 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.055600 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.056650 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.056886 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.059233 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:26.059244 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:26.059253 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:26.059316 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:26.064376 (+ 5060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.065680 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.065685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.065687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.065848 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.066228 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.066233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.066236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.066601 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:26.066783 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.067166 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.067421 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 23:28:26.067919 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.046893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.96.195:33933)
0620 23:28:26.050078 (+ 3185us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.050112 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.050401 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.050971 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.050985 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.051557 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:26.051875 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.051886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.053004 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.053014 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.055349 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.055370 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.056951 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.056956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.057172 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.058106 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:26.058125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:26.058928 (+ 803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:26.064605 (+ 5677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:26.064613 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:26.064619 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:26.065448 (+ 829us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:26.065973 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:26.065979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:26.065983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:26.066112 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:26.066715 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:26.066721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:26.066912 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.067052 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.067419 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2903,"thread_start_us":2731,"threads_started":1}
I20260620 23:28:26.084079 2494 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:37313
I20260620 23:28:26.084128 2629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:37313 every 8 connection(s)
I20260620 23:28:26.085847 2494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:26.088479 2630 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38901
I20260620 23:28:26.088840 2630 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:26.089299 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 2494
I20260620 23:28:26.089479 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:26.089854 2630 heartbeater.cc:511] Master 127.0.96.254:38901 requested a full tablet report, sending...
I20260620 23:28:26.089936 387 external_mini_cluster.cc:1428] Setting key 33acd2994466048a2435f21554d5cf5b
I20260620 23:28:26.091707 2143 ts_manager.cc:194] Registered new tserver with Master: e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:26.093087 2143 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:47481
I20260620 23:28:26.093647 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:28:26 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:28:26 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:26.146804 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.130814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.0.96.254:38901)
0620 23:28:26.131040 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.131047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.131547 (+ 500us) server_negotiation.cc:407] Connection header received
0620 23:28:26.131814 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.131818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.131885 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.131993 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:26.133570 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.134282 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.135564 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.135981 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.136119 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.140603 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:26.140608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:26.140612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:26.140652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:26.143763 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.144718 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.144721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.144724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.144859 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.145428 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.145431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.145433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.145712 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:26.145907 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.146341 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.146510 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:28:26.151839 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:26.159881 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:26.189684 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.176311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.0.1:56316)
0620 23:28:26.177479 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.177564 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.177886 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.179402 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.179411 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.179447 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:26.180030 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.180045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.182402 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.182408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.185230 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.185244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.187490 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.187496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.188721 (+ 1225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.188803 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.188966 (+ 163us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.189199 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1,"thread_start_us":751,"threads_started":1}
I20260620 23:28:26.190567 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.176482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.0.96.193:43395)
0620 23:28:26.178589 (+ 2107us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.178608 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.178636 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:28:26.178746 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.178752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.179104 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.180250 (+ 1146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:26.180523 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.182219 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.185499 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.187234 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.187395 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.187651 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.190013 (+ 2362us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.190240 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1944,"thread_start_us":1787,"threads_started":1}
I20260620 23:28:26.190733 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.176847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.0.1:60638)
0620 23:28:26.181582 (+ 4735us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.181605 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.181734 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.183591 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.183597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.183630 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:26.184152 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.184169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.186064 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.186070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.188172 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.188192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.189806 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.189814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.189981 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.190031 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.190135 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.190328 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4565,"mutex_wait_us":50,"thread_start_us":5541,"threads_started":1}
I20260620 23:28:26.191041 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.177159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.0.96.194:34037)
0620 23:28:26.182268 (+ 5109us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.182279 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.182317 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:28:26.182431 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.182444 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.182794 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.183347 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:26.184389 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.185880 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.188353 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.189548 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.189733 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.189977 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.190486 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.190702 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4924,"thread_start_us":4790,"threads_started":1}
I20260620 23:28:26.193178 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.178170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37313 (local address 127.0.0.1:42636)
0620 23:28:26.182737 (+ 4567us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.182763 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.182967 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.184040 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.184046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.184082 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:26.184586 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.184597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.187028 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.187038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.189368 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.189382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.192249 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.192256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.192435 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.192475 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.192581 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.192728 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4442,"mutex_wait_us":1,"thread_start_us":3122,"threads_started":1}
I20260620 23:28:26.193397 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.178501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.0.96.195:37313)
0620 23:28:26.181029 (+ 2528us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.181042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.182908 (+ 1866us) server_negotiation.cc:407] Connection header received
0620 23:28:26.183149 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.183181 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.183485 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.183698 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:26.184810 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.186883 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.189555 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.191740 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.191879 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.192084 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.192665 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.192948 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2278,"thread_start_us":2012,"threads_started":1}
I20260620 23:28:26.195510 2420 tablet_service.cc:1467] Processing CreateTablet for tablet cf75a607822947efb6147a91048defd6 (DEFAULT_TABLE table=test-table [id=66d78d445e9a456894484e653fd8fbdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:26.196926 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf75a607822947efb6147a91048defd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:26.197937 2560 tablet_service.cc:1467] Processing CreateTablet for tablet cf75a607822947efb6147a91048defd6 (DEFAULT_TABLE table=test-table [id=66d78d445e9a456894484e653fd8fbdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:26.198163 2280 tablet_service.cc:1467] Processing CreateTablet for tablet cf75a607822947efb6147a91048defd6 (DEFAULT_TABLE table=test-table [id=66d78d445e9a456894484e653fd8fbdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:26.199597 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf75a607822947efb6147a91048defd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:26.199667 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf75a607822947efb6147a91048defd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:26.212893 2649 tablet_bootstrap.cc:492] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Bootstrap starting.
I20260620 23:28:26.216611 2651 tablet_bootstrap.cc:492] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Bootstrap starting.
I20260620 23:28:26.219026 2649 tablet_bootstrap.cc:654] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:26.220304 2650 tablet_bootstrap.cc:492] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Bootstrap starting.
I20260620 23:28:26.220496 2651 tablet_bootstrap.cc:654] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:26.221112 2649 log.cc:824] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:26.221922 2651 log.cc:824] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:26.225356 2650 tablet_bootstrap.cc:654] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:26.226011 2649 tablet_bootstrap.cc:492] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: No bootstrap required, opened a new log
I20260620 23:28:26.226359 2649 ts_tablet_manager.cc:1397] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260620 23:28:26.227142 2650 log.cc:824] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:26.228106 2651 tablet_bootstrap.cc:492] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: No bootstrap required, opened a new log
I20260620 23:28:26.228464 2651 ts_tablet_manager.cc:1397] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 23:28:26.231411 2650 tablet_bootstrap.cc:492] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: No bootstrap required, opened a new log
I20260620 23:28:26.231801 2650 ts_tablet_manager.cc:1397] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260620 23:28:26.236429 2651 raft_consensus.cc:348] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.236857 2651 raft_consensus.cc:374] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:26.236969 2651 raft_consensus.cc:729] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Initialized, Role: FOLLOWER
I20260620 23:28:26.237684 2651 consensus_queue.cc:260] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.238117 2649 raft_consensus.cc:348] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.238765 2649 raft_consensus.cc:374] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:26.238890 2649 raft_consensus.cc:729] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Initialized, Role: FOLLOWER
I20260620 23:28:26.239748 2651 ts_tablet_manager.cc:1428] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Time spent starting tablet: real 0.011s user 0.013s sys 0.001s
I20260620 23:28:26.239686 2649 consensus_queue.cc:260] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.241176 2490 heartbeater.cc:503] Master 127.0.96.254:38901 was elected leader, sending a full tablet report...
I20260620 23:28:26.243829 2649 ts_tablet_manager.cc:1428] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 23:28:26.243767 2650 raft_consensus.cc:348] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.244366 2650 raft_consensus.cc:374] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:26.244486 2650 raft_consensus.cc:729] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9c36610194b32810b5eb1724258fd, State: Initialized, Role: FOLLOWER
I20260620 23:28:26.245304 2650 consensus_queue.cc:260] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.247530 2650 ts_tablet_manager.cc:1428] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260620 23:28:26.247566 2630 heartbeater.cc:503] Master 127.0.96.254:38901 was elected leader, sending a full tablet report...
W20260620 23:28:26.258656 2491 tablet.cc:2369] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:28:26.337855 2631 tablet.cc:2369] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:28:26.346568 2351 tablet.cc:2369] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:28:26.390812 2655 raft_consensus.cc:482] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:26.391155 2655 raft_consensus.cc:504] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.392935 2655 leader_election.cc:288] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:26.402298 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.393737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:34291 (local address 127.0.96.195:37313)
0620 23:28:26.393948 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.393953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.394981 (+ 1028us) server_negotiation.cc:407] Connection header received
0620 23:28:26.395102 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.395106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.395171 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.395262 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:26.395756 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.396796 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.399699 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.401042 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.401236 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.401324 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.401837 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.401988 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 23:28:26.402298 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.393690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:57375 (local address 127.0.96.194:34037)
0620 23:28:26.393927 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.393935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.394099 (+ 164us) server_negotiation.cc:407] Connection header received
0620 23:28:26.394344 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.394354 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.394478 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.394623 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:26.395369 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.396564 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.399699 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.401042 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.401247 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.401324 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.401850 (+ 526us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.401990 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:28:26.402513 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.393186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.96.193:57375)
0620 23:28:26.393948 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.393970 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.394139 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.394669 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.394680 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.394742 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:26.395169 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.395178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.396757 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.396761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.399492 (+ 2731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.399507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.401347 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.401351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.401603 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.401677 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.401817 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.401966 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":220,"thread_start_us":744,"threads_started":1}
I20260620 23:28:26.402642 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.393755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37313 (local address 127.0.96.193:34291)
0620 23:28:26.394811 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.394833 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.394975 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.395314 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.395320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.395343 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:26.395587 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.395593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.396949 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.396957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.399492 (+ 2535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.399507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.401335 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.401342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.401606 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.401687 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.401791 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.401966 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":468,"thread_start_us":494,"threads_started":1}
I20260620 23:28:26.403821 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75a607822947efb6147a91048defd6" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" is_pre_election: true
I20260620 23:28:26.403934 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75a607822947efb6147a91048defd6" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd" is_pre_election: true
I20260620 23:28:26.404443 2440 raft_consensus.cc:2454] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 0.
I20260620 23:28:26.404548 2585 raft_consensus.cc:2454] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 0.
I20260620 23:28:26.405301 2233 leader_election.cc:302] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4186cebe1f2e45a08fae117df36fbc02, df90a9c4b5714ae4a6d4dc1ec232749c; no voters:
I20260620 23:28:26.405736 2655 raft_consensus.cc:2781] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:26.405927 2655 raft_consensus.cc:482] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:26.406042 2655 raft_consensus.cc:3037] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:26.408605 2655 raft_consensus.cc:504] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.409441 2655 leader_election.cc:288] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 election: Requested vote from peers df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:26.410110 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75a607822947efb6147a91048defd6" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd"
I20260620 23:28:26.410311 2585 raft_consensus.cc:3037] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:26.410179 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75a607822947efb6147a91048defd6" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c"
I20260620 23:28:26.410576 2440 raft_consensus.cc:3037] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:26.412770 2585 raft_consensus.cc:2454] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 1.
I20260620 23:28:26.413254 2440 raft_consensus.cc:2454] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 1.
I20260620 23:28:26.413403 2232 leader_election.cc:302] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4186cebe1f2e45a08fae117df36fbc02, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:26.413962 2655 raft_consensus.cc:2781] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:26.414539 2655 raft_consensus.cc:686] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 LEADER]: Becoming Leader. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Running, Role: LEADER
I20260620 23:28:26.415179 2655 consensus_queue.cc:237] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.420924 2143 catalog_manager.cc:5495] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193). New cstate: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } health_report { overall_health: UNKNOWN } } }
Jun 20 23:28:26 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:26.525197 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.508507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.0.96.254:38901)
0620 23:28:26.508766 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.508780 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.509605 (+ 825us) server_negotiation.cc:407] Connection header received
0620 23:28:26.509934 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.509940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.510031 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.510181 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:26.511648 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.512831 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.514485 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.514948 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.515100 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.518990 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:26.518994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:26.518998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:26.519036 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:26.521826 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.522891 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.522898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.522906 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.523099 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:26.523480 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:26.523485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:26.523516 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:26.523831 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:26.523991 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.524429 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.524807 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":94}
I20260620 23:28:26.525244 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.508508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38901 (local address 127.0.0.1:35538)
0620 23:28:26.509448 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.509482 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.509726 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.510290 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.510294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.511124 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:26.511462 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.511472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.513079 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.513085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.514258 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.514271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.515177 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.515182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.515329 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.516068 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:26.516085 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:26.518746 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:26.522018 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:26.522032 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:26.522040 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:26.522742 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:26.523208 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:26.523215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:26.523219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:26.523369 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:26.523969 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:26.523978 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:26.524158 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.524328 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.524683 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":588,"threads_started":1}
I20260620 23:28:26.531670 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:35538:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:28:26.532912 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:26.541218 2420 tablet_service.cc:1467] Processing CreateTablet for tablet d5a4d599851144a49ecb5d2a8586ad54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d280b43d80934e34a66b41f9842c6084]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:28:26.541038 2280 tablet_service.cc:1467] Processing CreateTablet for tablet d5a4d599851144a49ecb5d2a8586ad54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d280b43d80934e34a66b41f9842c6084]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:28:26.542088 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a4d599851144a49ecb5d2a8586ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:26.542088 2420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a4d599851144a49ecb5d2a8586ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:26.541807 2560 tablet_service.cc:1467] Processing CreateTablet for tablet d5a4d599851144a49ecb5d2a8586ad54 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d280b43d80934e34a66b41f9842c6084]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:28:26.542748 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5a4d599851144a49ecb5d2a8586ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:26.549885 2651 tablet_bootstrap.cc:492] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: Bootstrap starting.
I20260620 23:28:26.550031 2650 tablet_bootstrap.cc:492] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: Bootstrap starting.
I20260620 23:28:26.550424 2649 tablet_bootstrap.cc:492] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: Bootstrap starting.
I20260620 23:28:26.551959 2650 tablet_bootstrap.cc:654] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:26.552002 2651 tablet_bootstrap.cc:654] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:26.552225 2649 tablet_bootstrap.cc:654] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:26.554076 2650 tablet_bootstrap.cc:492] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: No bootstrap required, opened a new log
I20260620 23:28:26.554251 2650 ts_tablet_manager.cc:1397] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260620 23:28:26.554991 2651 tablet_bootstrap.cc:492] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: No bootstrap required, opened a new log
I20260620 23:28:26.555182 2651 ts_tablet_manager.cc:1397] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260620 23:28:26.555053 2650 raft_consensus.cc:348] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.555369 2650 raft_consensus.cc:374] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:26.555465 2650 raft_consensus.cc:729] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9c36610194b32810b5eb1724258fd, State: Initialized, Role: FOLLOWER
I20260620 23:28:26.555567 2649 tablet_bootstrap.cc:492] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: No bootstrap required, opened a new log
I20260620 23:28:26.555714 2649 ts_tablet_manager.cc:1397] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:28:26.555689 2650 consensus_queue.cc:260] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.556280 2650 ts_tablet_manager.cc:1428] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:28:26.556470 2657 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.556522 2649 raft_consensus.cc:348] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.556813 2657 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.556830 2649 raft_consensus.cc:374] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:26.556931 2649 raft_consensus.cc:729] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Initialized, Role: FOLLOWER
I20260620 23:28:26.556228 2651 raft_consensus.cc:348] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.557235 2651 raft_consensus.cc:374] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:26.557232 2649 consensus_queue.cc:260] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.557379 2651 raft_consensus.cc:729] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Initialized, Role: FOLLOWER
I20260620 23:28:26.557673 2651 consensus_queue.cc:260] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.557910 2649 ts_tablet_manager.cc:1428] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 23:28:26.557823 2656 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.558269 2656 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.558327 2651 ts_tablet_manager.cc:1428] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:28:26.558655 2655 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.558924 2655 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.796767 2655 raft_consensus.cc:482] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:26.797122 2655 raft_consensus.cc:504] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.797931 2655 leader_election.cc:288] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:26.798801 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a4d599851144a49ecb5d2a8586ad54" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" is_pre_election: true
I20260620 23:28:26.798830 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a4d599851144a49ecb5d2a8586ad54" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd" is_pre_election: true
I20260620 23:28:26.799147 2585 raft_consensus.cc:2454] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 0.
I20260620 23:28:26.799201 2440 raft_consensus.cc:2454] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 0.
I20260620 23:28:26.799706 2232 leader_election.cc:302] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4186cebe1f2e45a08fae117df36fbc02, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:26.800107 2660 raft_consensus.cc:2781] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:26.800235 2660 raft_consensus.cc:482] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:26.800379 2660 raft_consensus.cc:3037] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:26.800997 2655 consensus_queue.cc:1035] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:26.803208 2660 raft_consensus.cc:504] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.803994 2660 leader_election.cc:288] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 election: Requested vote from peers df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:26.804605 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a4d599851144a49ecb5d2a8586ad54" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c"
I20260620 23:28:26.804720 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5a4d599851144a49ecb5d2a8586ad54" candidate_uuid: "4186cebe1f2e45a08fae117df36fbc02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd"
I20260620 23:28:26.804884 2440 raft_consensus.cc:3037] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:26.804999 2585 raft_consensus.cc:3037] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:26.807823 2440 raft_consensus.cc:2454] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 1.
I20260620 23:28:26.808192 2585 raft_consensus.cc:2454] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4186cebe1f2e45a08fae117df36fbc02 in term 1.
I20260620 23:28:26.808307 2233 leader_election.cc:302] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4186cebe1f2e45a08fae117df36fbc02, df90a9c4b5714ae4a6d4dc1ec232749c; no voters:
I20260620 23:28:26.808743 2660 raft_consensus.cc:2781] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:26.809324 2660 raft_consensus.cc:686] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 LEADER]: Becoming Leader. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Running, Role: LEADER
I20260620 23:28:26.809695 2660 consensus_queue.cc:237] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:26.812785 2655 consensus_queue.cc:1035] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:26.821905 2673 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: TxnStatusTablet state changed. Reason: New leader 4186cebe1f2e45a08fae117df36fbc02. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.822381 2673 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:28:26.823374 2141 catalog_manager.cc:5495] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193). New cstate: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } health_report { overall_health: UNKNOWN } } }
I20260620 23:28:26.832384 2679 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:28:26.833576 2679 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:28:26.866066 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.859154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.0.1:43802)
0620 23:28:26.859671 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.859698 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.859912 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.860534 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.860539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.860551 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:26.860814 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.860828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.862379 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.862385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.863379 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.863390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.864167 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.864172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.864336 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.864396 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:26.865488 (+ 1092us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:26.865507 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.865594 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.865756 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289}
I20260620 23:28:26.866175 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.859209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.0.96.193:43395)
0620 23:28:26.860130 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.860137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.860168 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:28:26.860261 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.860265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.860338 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.860515 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:26.861044 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.862223 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.863528 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.863993 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.864157 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.864639 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:26.865373 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:26.865526 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.865634 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.865721 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":441,"threads_started":1}
I20260620 23:28:26.873656 2585 raft_consensus.cc:1261] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Refusing update from remote peer 4186cebe1f2e45a08fae117df36fbc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:26.873827 2440 raft_consensus.cc:1261] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Refusing update from remote peer 4186cebe1f2e45a08fae117df36fbc02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:26.874477 2676 consensus_queue.cc:1035] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:26.875093 2674 consensus_queue.cc:1035] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:26.888018 2656 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: TxnStatusTablet state changed. Reason: New leader 4186cebe1f2e45a08fae117df36fbc02. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.888382 2656 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.888927 2657 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: TxnStatusTablet state changed. Reason: New leader 4186cebe1f2e45a08fae117df36fbc02. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.889216 2657 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.892048 2674 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.892460 2674 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:28:26.892379 2676 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.892688 2676 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P 4186cebe1f2e45a08fae117df36fbc02: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:28:26.893891 2675 mvcc.cc:205] Tried to move back new op lower bound from 7299064245745037312 to 7299064245496651776. Current Snapshot: MvccSnapshot[applied={T|T < 7299064245745037312}]
I20260620 23:28:26.898566 2657 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.898836 2657 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P e4c9c36610194b32810b5eb1724258fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.904322 2656 tablet_replica.cc:439] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4186cebe1f2e45a08fae117df36fbc02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } } }
I20260620 23:28:26.904582 2656 tablet_replica.cc:442] T d5a4d599851144a49ecb5d2a8586ad54 P df90a9c4b5714ae4a6d4dc1ec232749c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:28:26.916674 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.910368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.0.96.193:43395)
0620 23:28:26.910549 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.910554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.911329 (+ 775us) server_negotiation.cc:407] Connection header received
0620 23:28:26.911566 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.911571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.911650 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.911756 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:26.912157 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.912920 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.914489 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.915120 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.915310 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.915913 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:26.916036 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:26.916137 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.916285 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.916373 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 23:28:26.939049 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.931720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:37123 (local address 127.0.96.193:43395)
0620 23:28:26.932184 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:26.932195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:26.933062 (+ 867us) server_negotiation.cc:407] Connection header received
0620 23:28:26.933361 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:26.933366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:26.933492 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:26.933595 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:26.934391 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.935181 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.936974 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.937580 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.937737 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.938189 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:26.938299 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:26.938403 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:26.938530 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 23:28:26.938641 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":262}
I20260620 23:28:26.939123 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:26.931619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.96.193:37123)
0620 23:28:26.932877 (+ 1258us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:26.932908 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:26.933206 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:26.933624 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:26.933629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:26.933640 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:26.934146 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.934161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.935372 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.935377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.936650 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:26.936664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:26.937755 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:26.937761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:26.937895 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:26.937963 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:26.938401 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:26.938412 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:28:26.938512 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:28:26.938729 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":40,"thread_start_us":790,"threads_started":1}
I20260620 23:28:27.021400 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:27.021703 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:27.026084 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Sending DeleteTablet for 3 replicas of tablet cf75a607822947efb6147a91048defd6
I20260620 23:28:27.027459 2280 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75a607822947efb6147a91048defd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:27 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:56316
I20260620 23:28:27.027604 2420 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75a607822947efb6147a91048defd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:27 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:60638
I20260620 23:28:27.027719 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75a607822947efb6147a91048defd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:27 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42636
I20260620 23:28:27.030462 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:27.032158 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:27.044041 2419 tablet_service.cc:1467] Processing CreateTablet for tablet f620f84c23ed4f37ad1dd120b8c840b0 (DEFAULT_TABLE table=test-table [id=80868e8f61104747b7e24be367115ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:27.044173 2559 tablet_service.cc:1467] Processing CreateTablet for tablet f620f84c23ed4f37ad1dd120b8c840b0 (DEFAULT_TABLE table=test-table [id=80868e8f61104747b7e24be367115ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:27.044847 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f620f84c23ed4f37ad1dd120b8c840b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:27.044912 2559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f620f84c23ed4f37ad1dd120b8c840b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:27.048261 2697 tablet_replica.cc:330] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: stopping tablet replica
I20260620 23:28:27.048738 2697 raft_consensus.cc:2229] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:27.049959 2279 tablet_service.cc:1467] Processing CreateTablet for tablet f620f84c23ed4f37ad1dd120b8c840b0 (DEFAULT_TABLE table=test-table [id=80868e8f61104747b7e24be367115ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:27.050531 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f620f84c23ed4f37ad1dd120b8c840b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:27.053324 2649 tablet_bootstrap.cc:492] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Bootstrap starting.
I20260620 23:28:27.053579 2650 tablet_bootstrap.cc:492] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Bootstrap starting.
I20260620 23:28:27.055315 2650 tablet_bootstrap.cc:654] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:27.055510 2649 tablet_bootstrap.cc:654] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:27.060706 2649 tablet_bootstrap.cc:492] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: No bootstrap required, opened a new log
I20260620 23:28:27.061158 2649 ts_tablet_manager.cc:1397] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 23:28:27.061331 2650 tablet_bootstrap.cc:492] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: No bootstrap required, opened a new log
I20260620 23:28:27.061574 2650 ts_tablet_manager.cc:1397] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 23:28:27.062422 2649 raft_consensus.cc:348] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.062435 2650 raft_consensus.cc:348] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.062718 2650 raft_consensus.cc:374] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:27.062825 2650 raft_consensus.cc:729] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9c36610194b32810b5eb1724258fd, State: Initialized, Role: FOLLOWER
I20260620 23:28:27.062865 2649 raft_consensus.cc:374] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:27.063055 2649 raft_consensus.cc:729] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Initialized, Role: FOLLOWER
I20260620 23:28:27.063169 2650 consensus_queue.cc:260] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.063515 2649 consensus_queue.cc:260] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.064160 2649 ts_tablet_manager.cc:1428] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 23:28:27.066217 2697 raft_consensus.cc:2258] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:27.068707 2697 ts_tablet_manager.cc:1905] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:27.071223 2696 tablet_replica.cc:330] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: stopping tablet replica
I20260620 23:28:27.071997 2696 raft_consensus.cc:2229] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:27.072577 2696 raft_consensus.cc:2258] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:27.073700 2650 ts_tablet_manager.cc:1428] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20260620 23:28:27.074023 2695 tablet_replica.cc:330] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: stopping tablet replica
I20260620 23:28:27.074517 2695 raft_consensus.cc:2229] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:27.074774 2656 raft_consensus.cc:482] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:27.074867 2695 raft_consensus.cc:2258] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:27.074997 2656 raft_consensus.cc:504] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.076969 2695 ts_tablet_manager.cc:1905] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:27.077108 2656 leader_election.cc:288] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:27.082507 2697 ts_tablet_manager.cc:1918] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:27.082913 2700 tablet_bootstrap.cc:492] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Bootstrap starting.
I20260620 23:28:27.085342 2700 tablet_bootstrap.cc:654] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:27.086201 2695 ts_tablet_manager.cc:1918] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:27.086617 2695 log.cc:1196] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/cf75a607822947efb6147a91048defd6
I20260620 23:28:27.086674 2697 log.cc:1196] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/cf75a607822947efb6147a91048defd6
I20260620 23:28:27.087321 2697 ts_tablet_manager.cc:1939] T cf75a607822947efb6147a91048defd6 P 4186cebe1f2e45a08fae117df36fbc02: Deleting consensus metadata
I20260620 23:28:27.087426 2695 ts_tablet_manager.cc:1939] T cf75a607822947efb6147a91048defd6 P e4c9c36610194b32810b5eb1724258fd: Deleting consensus metadata
I20260620 23:28:27.089568 2129 catalog_manager.cc:4841] TS 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395): tablet cf75a607822947efb6147a91048defd6 (table test-table [id=66d78d445e9a456894484e653fd8fbdb]) successfully deleted
I20260620 23:28:27.089941 2129 catalog_manager.cc:4841] TS e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313): tablet cf75a607822947efb6147a91048defd6 (table test-table [id=66d78d445e9a456894484e653fd8fbdb]) successfully deleted
I20260620 23:28:27.091056 2696 ts_tablet_manager.cc:1905] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:27.092867 2657 raft_consensus.cc:482] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:27.093125 2657 raft_consensus.cc:504] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.095335 2657 leader_election.cc:288] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037)
I20260620 23:28:27.120126 2700 tablet_bootstrap.cc:492] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: No bootstrap required, opened a new log
I20260620 23:28:27.120437 2700 ts_tablet_manager.cc:1397] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Time spent bootstrapping tablet: real 0.054s user 0.019s sys 0.012s
I20260620 23:28:27.121517 2700 raft_consensus.cc:348] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.121930 2700 raft_consensus.cc:374] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:27.122081 2700 raft_consensus.cc:729] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Initialized, Role: FOLLOWER
I20260620 23:28:27.122486 2700 consensus_queue.cc:260] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.123459 2700 ts_tablet_manager.cc:1428] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:28:27.125363 2696 ts_tablet_manager.cc:1918] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:27.128384 2696 log.cc:1196] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/cf75a607822947efb6147a91048defd6
I20260620 23:28:27.133623 2696 ts_tablet_manager.cc:1939] T cf75a607822947efb6147a91048defd6 P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting consensus metadata
I20260620 23:28:27.136314 2130 catalog_manager.cc:4841] TS df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037): tablet cf75a607822947efb6147a91048defd6 (table test-table [id=66d78d445e9a456894484e653fd8fbdb]) successfully deleted
I20260620 23:28:27.152894 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.096471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.96.195:38115)
0620 23:28:27.097972 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:27.098004 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:27.098201 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:27.132881 (+ 34680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:27.132886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:27.132916 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:27.133592 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.133611 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.148038 (+ 14427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.148044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.150141 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.150154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.152048 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.152055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.152244 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.152286 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:28:27.152365 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:28:27.152519 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1322,"thread_start_us":55758,"threads_started":1}
I20260620 23:28:27.156057 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.096789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38115 (local address 127.0.96.193:43395)
0620 23:28:27.125425 (+ 28636us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.125431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.125463 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:28:27.132580 (+ 7117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.132586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.132669 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.133333 (+ 664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:27.134546 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.135904 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.150362 (+ 14458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.151805 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.155373 (+ 3568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.155442 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.155514 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.155653 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28474,"thread_start_us":28757,"threads_started":1}
I20260620 23:28:27.157367 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.097629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:43025 (local address 127.0.96.194:34037)
0620 23:28:27.099063 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.099075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.132278 (+ 33203us) server_negotiation.cc:407] Connection header received
0620 23:28:27.141180 (+ 8902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.141186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.141267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.141330 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:27.143491 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.144698 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.154451 (+ 9753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.155734 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.155923 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.155994 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.156790 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.156943 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":1238,"thread_start_us":888,"threads_started":1}
I20260620 23:28:27.157811 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.097324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.96.195:43025)
0620 23:28:27.132169 (+ 34845us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:27.132195 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:27.136127 (+ 3932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:27.141593 (+ 5466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:27.141598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:27.141624 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:27.142035 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.142047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.144876 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.144881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.147182 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.147198 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.156363 (+ 9165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.156375 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.156591 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.156635 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:28:27.157318 (+ 683us) client_negotiation.cc:240] Negotiation successful
0620 23:28:27.157480 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34675,"thread_start_us":2124,"threads_started":1}
I20260620 23:28:27.157596 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "e4c9c36610194b32810b5eb1724258fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02" is_pre_election: true
I20260620 23:28:27.158207 2300 raft_consensus.cc:2454] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c9c36610194b32810b5eb1724258fd in term 0.
I20260620 23:28:27.158794 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.077906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:52015 (local address 127.0.96.193:43395)
0620 23:28:27.078124 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.078130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.095756 (+ 17626us) server_negotiation.cc:407] Connection header received
0620 23:28:27.095885 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.095893 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.095994 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.096084 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:27.136770 (+ 40686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.138228 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.153115 (+ 14887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.154489 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.154697 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.154771 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.158433 (+ 3662us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.158572 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:28:27.159565 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.077486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.96.194:52015)
0620 23:28:27.094456 (+ 16970us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:27.094480 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:27.094640 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:27.127481 (+ 32841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:27.127487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:27.127534 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:27.128045 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.128063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.138503 (+ 10440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.138511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.151984 (+ 13473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.152017 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.155042 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.155049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.158168 (+ 3119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.158205 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:28:27.158994 (+ 789us) client_negotiation.cc:240] Negotiation successful
0620 23:28:27.159136 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16813,"mutex_wait_us":1,"thread_start_us":17238,"threads_started":1}
I20260620 23:28:27.159610 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "e4c9c36610194b32810b5eb1724258fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" is_pre_election: true
I20260620 23:28:27.159912 2440 raft_consensus.cc:2454] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4c9c36610194b32810b5eb1724258fd in term 0.
I20260620 23:28:27.160187 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02" is_pre_election: true
I20260620 23:28:27.160439 2300 raft_consensus.cc:2454] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 0.
I20260620 23:28:27.160741 2513 leader_election.cc:302] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df90a9c4b5714ae4a6d4dc1ec232749c, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:27.161087 2372 leader_election.cc:302] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4186cebe1f2e45a08fae117df36fbc02, df90a9c4b5714ae4a6d4dc1ec232749c; no voters:
I20260620 23:28:27.161314 2657 raft_consensus.cc:2781] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:27.161425 2657 raft_consensus.cc:482] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:27.161532 2657 raft_consensus.cc:3037] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:27.161614 2656 raft_consensus.cc:2781] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:27.161731 2656 raft_consensus.cc:482] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:27.161823 2656 raft_consensus.cc:3037] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:27.164822 2657 raft_consensus.cc:504] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.164839 2656 raft_consensus.cc:504] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.165455 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.095161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37313 (local address 127.0.96.194:43423)
0620 23:28:27.147728 (+ 52567us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:27.147748 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:27.147869 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:27.157379 (+ 9510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:27.157388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:27.157443 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:27.157947 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.157961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.159727 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.159736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.162124 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.162143 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.164543 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.164559 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.164732 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.164766 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:28:27.164869 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 23:28:27.165060 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52423,"thread_start_us":53809,"threads_started":1}
I20260620 23:28:27.165560 2657 leader_election.cc:288] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [CANDIDATE]: Term 1 election: Requested vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037)
I20260620 23:28:27.166041 2440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "e4c9c36610194b32810b5eb1724258fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c"
I20260620 23:28:27.166224 2656 leader_election.cc:288] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 election: Requested vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:27.166249 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.152573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:43423 (local address 127.0.96.195:37313)
0620 23:28:27.156780 (+ 4207us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.156786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.156806 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:28:27.156859 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.156864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.156940 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.157080 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:27.158250 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.159526 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.162443 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.164044 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.164269 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.164356 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.165735 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.165904 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4053,"thread_start_us":3885,"threads_started":1}
I20260620 23:28:27.166257 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "e4c9c36610194b32810b5eb1724258fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02"
I20260620 23:28:27.166376 2440 raft_consensus.cc:2379] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e4c9c36610194b32810b5eb1724258fd in current term 1: Already voted for candidate df90a9c4b5714ae4a6d4dc1ec232749c in this term.
I20260620 23:28:27.166499 2300 raft_consensus.cc:3037] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:27.167253 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd" is_pre_election: true
I20260620 23:28:27.167621 2585 raft_consensus.cc:2379] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df90a9c4b5714ae4a6d4dc1ec232749c in current term 1: Already voted for candidate e4c9c36610194b32810b5eb1724258fd in this term.
I20260620 23:28:27.168525 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd"
I20260620 23:28:27.168800 2585 raft_consensus.cc:2379] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df90a9c4b5714ae4a6d4dc1ec232749c in current term 1: Already voted for candidate e4c9c36610194b32810b5eb1724258fd in this term.
I20260620 23:28:27.169556 2299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f620f84c23ed4f37ad1dd120b8c840b0" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02"
I20260620 23:28:27.170521 2300 raft_consensus.cc:2454] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4c9c36610194b32810b5eb1724258fd in term 1.
I20260620 23:28:27.170492 2372 leader_election.cc:302] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df90a9c4b5714ae4a6d4dc1ec232749c; no voters: 4186cebe1f2e45a08fae117df36fbc02, e4c9c36610194b32810b5eb1724258fd
I20260620 23:28:27.170956 2656 raft_consensus.cc:2726] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 23:28:27.170981 2512 leader_election.cc:302] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4186cebe1f2e45a08fae117df36fbc02, e4c9c36610194b32810b5eb1724258fd; no voters: df90a9c4b5714ae4a6d4dc1ec232749c
I20260620 23:28:27.171406 2657 raft_consensus.cc:2781] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:27.171999 2657 raft_consensus.cc:686] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 LEADER]: Becoming Leader. State: Replica: e4c9c36610194b32810b5eb1724258fd, State: Running, Role: LEADER
I20260620 23:28:27.172513 2657 consensus_queue.cc:237] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:27.177397 2143 catalog_manager.cc:5495] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd reported cstate change: term changed from 0 to 1, leader changed from <none> to e4c9c36610194b32810b5eb1724258fd (127.0.96.195). New cstate: current_term: 1 leader_uuid: "e4c9c36610194b32810b5eb1724258fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } health_report { overall_health: HEALTHY } } }
I20260620 23:28:27.228657 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.221340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.0.96.195:37313)
0620 23:28:27.221583 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.221589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.221609 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:28:27.221742 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.221749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.221866 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.222149 (+ 283us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:27.222563 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.223619 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.225656 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.226151 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.226280 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.226869 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:27.227701 (+ 832us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:27.227885 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.228161 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.228260 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:28:27.250502 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.242683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.96.195:48329)
0620 23:28:27.243620 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:27.243644 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:27.243825 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:27.244298 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:27.244301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:27.244312 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:27.244676 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.244685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.246246 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.246251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.247500 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.247516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.248847 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.248860 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.249216 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.249335 (+ 119us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:27.249907 (+ 572us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:27.249917 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 23:28:27.249992 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 23:28:27.250144 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":461,"threads_started":1}
I20260620 23:28:27.250504 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.242688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:48329 (local address 127.0.96.193:43395)
0620 23:28:27.242949 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.242957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.243785 (+ 828us) server_negotiation.cc:407] Connection header received
0620 23:28:27.243997 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.244004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.244109 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.244252 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:27.244853 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.246041 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.247693 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.248433 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.248605 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.249588 (+ 983us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:27.249771 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:27.249897 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.250020 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.250127 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:28:27.262241 2440 raft_consensus.cc:1261] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Refusing update from remote peer e4c9c36610194b32810b5eb1724258fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:27.262253 2300 raft_consensus.cc:1261] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Refusing update from remote peer e4c9c36610194b32810b5eb1724258fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:27.263083 2709 consensus_queue.cc:1035] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:27.263808 2657 consensus_queue.cc:1035] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:27.301433 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.295311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:58897 (local address 127.0.96.195:37313)
0620 23:28:27.295606 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:27.295618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:27.295638 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:28:27.295707 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:27.295711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:27.295780 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:27.295899 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:27.296569 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.297659 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.299372 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.299833 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.299976 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.300489 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:27.300669 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:27.300791 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:27.300968 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 23:28:27.301097 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 23:28:27.301497 2693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:27.295216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37313 (local address 127.0.96.193:58897)
0620 23:28:27.295422 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:27.295452 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:27.295591 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:27.296102 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:27.296107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:27.296122 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:27.296384 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.296399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.297795 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.297802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.299186 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:27.299200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:27.300005 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:27.300012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:27.300196 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:27.300253 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:27.300810 (+ 557us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:27.300824 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:28:27.300922 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:28:27.301129 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:28:27.302995 2675 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:28:28.928920 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:28.929323 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:28.935026 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Sending DeleteTablet for 3 replicas of tablet f620f84c23ed4f37ad1dd120b8c840b0
I20260620 23:28:28.936205 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet f620f84c23ed4f37ad1dd120b8c840b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:28 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42636
I20260620 23:28:28.936295 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet f620f84c23ed4f37ad1dd120b8c840b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:28 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:60638
I20260620 23:28:28.936916 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet f620f84c23ed4f37ad1dd120b8c840b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:28 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:56316
I20260620 23:28:28.948777 2741 tablet_replica.cc:330] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: stopping tablet replica
I20260620 23:28:28.949308 2741 raft_consensus.cc:2229] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:28.949744 2741 raft_consensus.cc:2258] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:28.951376 2741 ts_tablet_manager.cc:1905] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:28.958003 2742 tablet_replica.cc:330] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: stopping tablet replica
I20260620 23:28:28.958222 2740 tablet_replica.cc:330] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: stopping tablet replica
I20260620 23:28:28.958354 2742 raft_consensus.cc:2229] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:28.958530 2740 raft_consensus.cc:2229] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:28.958791 2742 raft_consensus.cc:2258] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:28.959638 2740 raft_consensus.cc:2258] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:28.960202 2742 ts_tablet_manager.cc:1905] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:28.961315 2740 ts_tablet_manager.cc:1905] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:28.961477 2741 ts_tablet_manager.cc:1918] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:28:28.961766 2741 log.cc:1196] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/f620f84c23ed4f37ad1dd120b8c840b0
I20260620 23:28:28.962414 2741 ts_tablet_manager.cc:1939] T f620f84c23ed4f37ad1dd120b8c840b0 P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting consensus metadata
I20260620 23:28:28.963682 2130 catalog_manager.cc:4841] TS df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037): tablet f620f84c23ed4f37ad1dd120b8c840b0 (table test-table [id=80868e8f61104747b7e24be367115ccb]) successfully deleted
I20260620 23:28:28.966914 2742 ts_tablet_manager.cc:1918] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:28:28.967062 2742 log.cc:1196] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/f620f84c23ed4f37ad1dd120b8c840b0
I20260620 23:28:28.967512 2742 ts_tablet_manager.cc:1939] T f620f84c23ed4f37ad1dd120b8c840b0 P 4186cebe1f2e45a08fae117df36fbc02: Deleting consensus metadata
I20260620 23:28:28.968920 2129 catalog_manager.cc:4841] TS 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395): tablet f620f84c23ed4f37ad1dd120b8c840b0 (table test-table [id=80868e8f61104747b7e24be367115ccb]) successfully deleted
I20260620 23:28:28.970986 2740 ts_tablet_manager.cc:1918] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:28:28.971149 2740 log.cc:1196] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/f620f84c23ed4f37ad1dd120b8c840b0
I20260620 23:28:28.971683 2740 ts_tablet_manager.cc:1939] T f620f84c23ed4f37ad1dd120b8c840b0 P e4c9c36610194b32810b5eb1724258fd: Deleting consensus metadata
I20260620 23:28:28.973141 2129 catalog_manager.cc:4841] TS e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313): tablet f620f84c23ed4f37ad1dd120b8c840b0 (table test-table [id=80868e8f61104747b7e24be367115ccb]) successfully deleted
Jun 20 23:28:28 dist-test-slave-v34r krb5kdc[2098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:28:29 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:29.012535 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:28.996104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.0.96.254:38901)
0620 23:28:28.996794 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:28.996799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:28.996826 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:28:28.996901 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:28.996905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:28.997081 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:28.997234 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:28.998715 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:28.999602 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.001075 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.001568 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.001710 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:29.006358 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:29.006366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:29.006371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:29.006427 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:29.009699 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:29.010476 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:29.010481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:29.010483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:29.010641 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:29.011064 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:29.011069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:29.011071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:29.011416 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:29.011631 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:29.011959 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 23:28:29.012210 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":362,"threads_started":1}
I20260620 23:28:29.017514 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:29.018226 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:29.026160 2279 tablet_service.cc:1467] Processing CreateTablet for tablet 393e31f49ea041eba473c7c266f1a95e (DEFAULT_TABLE table=test-table [id=66df09f4149b432c901bdea525acd61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:29.026402 2419 tablet_service.cc:1467] Processing CreateTablet for tablet 393e31f49ea041eba473c7c266f1a95e (DEFAULT_TABLE table=test-table [id=66df09f4149b432c901bdea525acd61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:29.026773 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393e31f49ea041eba473c7c266f1a95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:29.026935 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393e31f49ea041eba473c7c266f1a95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:29.033107 2751 tablet_bootstrap.cc:492] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Bootstrap starting.
I20260620 23:28:29.033550 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 393e31f49ea041eba473c7c266f1a95e (DEFAULT_TABLE table=test-table [id=66df09f4149b432c901bdea525acd61c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:29.033782 2752 tablet_bootstrap.cc:492] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Bootstrap starting.
I20260620 23:28:29.034103 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 393e31f49ea041eba473c7c266f1a95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:29.034865 2751 tablet_bootstrap.cc:654] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:29.035806 2752 tablet_bootstrap.cc:654] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:29.037385 2751 tablet_bootstrap.cc:492] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: No bootstrap required, opened a new log
I20260620 23:28:29.037647 2751 ts_tablet_manager.cc:1397] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 23:28:29.038429 2751 raft_consensus.cc:348] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.038640 2751 raft_consensus.cc:374] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:29.038748 2751 raft_consensus.cc:729] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Initialized, Role: FOLLOWER
I20260620 23:28:29.039016 2751 consensus_queue.cc:260] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.039876 2751 ts_tablet_manager.cc:1428] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:28:29.043903 2752 tablet_bootstrap.cc:492] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: No bootstrap required, opened a new log
I20260620 23:28:29.044082 2752 ts_tablet_manager.cc:1397] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260620 23:28:29.044967 2756 tablet_bootstrap.cc:492] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Bootstrap starting.
I20260620 23:28:29.044981 2752 raft_consensus.cc:348] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.045253 2752 raft_consensus.cc:374] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:29.045354 2752 raft_consensus.cc:729] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Initialized, Role: FOLLOWER
I20260620 23:28:29.045712 2752 consensus_queue.cc:260] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.046360 2752 ts_tablet_manager.cc:1428] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 23:28:29.047024 2756 tablet_bootstrap.cc:654] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:29.050585 2756 tablet_bootstrap.cc:492] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: No bootstrap required, opened a new log
I20260620 23:28:29.050776 2756 ts_tablet_manager.cc:1397] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260620 23:28:29.051644 2756 raft_consensus.cc:348] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.051916 2756 raft_consensus.cc:374] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:29.052016 2756 raft_consensus.cc:729] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9c36610194b32810b5eb1724258fd, State: Initialized, Role: FOLLOWER
I20260620 23:28:29.052294 2756 consensus_queue.cc:260] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.052865 2756 ts_tablet_manager.cc:1428] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:28:29.202749 2755 raft_consensus.cc:482] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:29.203002 2755 raft_consensus.cc:504] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.203760 2755 leader_election.cc:288] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:29.204873 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393e31f49ea041eba473c7c266f1a95e" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02" is_pre_election: true
I20260620 23:28:29.204844 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393e31f49ea041eba473c7c266f1a95e" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd" is_pre_election: true
I20260620 23:28:29.205237 2300 raft_consensus.cc:2454] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 0.
I20260620 23:28:29.205264 2585 raft_consensus.cc:2454] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 0.
I20260620 23:28:29.205879 2372 leader_election.cc:302] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df90a9c4b5714ae4a6d4dc1ec232749c, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:29.206210 2755 raft_consensus.cc:2781] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:29.206328 2755 raft_consensus.cc:482] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:29.206485 2755 raft_consensus.cc:3037] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:29.208611 2755 raft_consensus.cc:504] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.209245 2755 leader_election.cc:288] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 election: Requested vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:29.210184 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393e31f49ea041eba473c7c266f1a95e" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd"
I20260620 23:28:29.210199 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "393e31f49ea041eba473c7c266f1a95e" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02"
I20260620 23:28:29.210513 2300 raft_consensus.cc:3037] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:29.210539 2585 raft_consensus.cc:3037] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:29.213976 2300 raft_consensus.cc:2454] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 1.
I20260620 23:28:29.213946 2585 raft_consensus.cc:2454] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 1.
I20260620 23:28:29.214646 2372 leader_election.cc:302] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df90a9c4b5714ae4a6d4dc1ec232749c, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:29.215055 2755 raft_consensus.cc:2781] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:29.215687 2755 raft_consensus.cc:686] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 LEADER]: Becoming Leader. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Running, Role: LEADER
I20260620 23:28:29.216148 2755 consensus_queue.cc:237] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.220228 2143 catalog_manager.cc:5495] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c reported cstate change: term changed from 0 to 1, leader changed from <none> to df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194). New cstate: current_term: 1 leader_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } health_report { overall_health: UNKNOWN } } }
I20260620 23:28:29.247936 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:29.241814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.0.96.194:34037)
0620 23:28:29.242569 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:29.242574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:29.242596 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:28:29.242724 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:29.242732 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:29.242872 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:29.243079 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:29.243488 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.244262 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.245682 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.246099 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.246230 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:29.246670 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:29.247282 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:29.247451 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:29.247512 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:28:29.247616 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":379,"threads_started":1}
I20260620 23:28:29.268179 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:29.260756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.96.194:39829)
0620 23:28:29.261654 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:29.261672 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:29.261797 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:29.262551 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:29.262555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:29.262565 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:29.263234 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:29.263244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.264448 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.264451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:29.265719 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:29.265741 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.267046 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.267051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:29.267177 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:29.267242 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:29.267716 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:29.267724 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:28:29.267795 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 23:28:29.267905 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":484,"threads_started":1}
I20260620 23:28:29.268378 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:29.261124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:39829 (local address 127.0.96.193:43395)
0620 23:28:29.261950 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:29.261955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:29.261982 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:28:29.262063 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:29.262067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:29.262144 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:29.262271 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:29.263434 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.264270 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.266000 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.266795 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.266935 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:29.267475 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:29.267615 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:29.267702 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:29.267891 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:28:29.267981 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":487,"threads_started":1}
I20260620 23:28:29.280058 2300 raft_consensus.cc:1261] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Refusing update from remote peer df90a9c4b5714ae4a6d4dc1ec232749c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:29.280120 2585 raft_consensus.cc:1261] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Refusing update from remote peer df90a9c4b5714ae4a6d4dc1ec232749c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:29.280874 2755 consensus_queue.cc:1035] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:29.281358 2758 consensus_queue.cc:1035] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:29.287927 2723 mvcc.cc:205] Tried to move back new op lower bound from 7299064255603384320 to 7299064255354109952. Current Snapshot: MvccSnapshot[applied={T|T < 7299064255603384320}]
I20260620 23:28:29.314745 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:29.307927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:47135 (local address 127.0.96.194:34037)
0620 23:28:29.308090 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:29.308094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:29.308443 (+ 349us) server_negotiation.cc:407] Connection header received
0620 23:28:29.308667 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:29.308674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:29.308754 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:29.308852 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:29.309544 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.310561 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.312251 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.312664 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.312838 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:29.313882 (+ 1044us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:29.314065 (+ 183us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:29.314170 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:29.314358 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 23:28:29.314442 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:28:29.315338 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:29.307668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.96.193:47135)
0620 23:28:29.308306 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:29.308323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:29.308507 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:29.308877 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:29.308882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:29.308894 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:28:29.309280 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:29.309300 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.310760 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.310766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:29.312067 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:29.312083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:29.312900 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:29.312905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:29.313153 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:29.313219 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:28:29.314178 (+ 959us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:28:29.314190 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:28:29.314308 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:28:29.314446 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":367,"threads_started":1}
I20260620 23:28:29.358551 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:29.358927 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:29.363173 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Sending DeleteTablet for 3 replicas of tablet 393e31f49ea041eba473c7c266f1a95e
I20260620 23:28:29.364250 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet 393e31f49ea041eba473c7c266f1a95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:29 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:60638
I20260620 23:28:29.364635 2741 tablet_replica.cc:330] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: stopping tablet replica
I20260620 23:28:29.364584 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet 393e31f49ea041eba473c7c266f1a95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:29 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:56316
I20260620 23:28:29.364615 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 393e31f49ea041eba473c7c266f1a95e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:29 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42636
I20260620 23:28:29.364863 2742 tablet_replica.cc:330] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: stopping tablet replica
I20260620 23:28:29.364876 2741 raft_consensus.cc:2229] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:29.365018 2740 tablet_replica.cc:330] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: stopping tablet replica
I20260620 23:28:29.365072 2742 raft_consensus.cc:2229] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:29.365283 2742 raft_consensus.cc:2258] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:29.365995 2741 raft_consensus.cc:2258] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:29.366012 2740 raft_consensus.cc:2229] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:29.366261 2740 raft_consensus.cc:2258] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:29.367494 2741 ts_tablet_manager.cc:1905] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:29.367846 2740 ts_tablet_manager.cc:1905] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:29.368482 2742 ts_tablet_manager.cc:1905] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:29.373545 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:29.374460 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:29.375180 2742 ts_tablet_manager.cc:1918] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:29.375357 2742 log.cc:1196] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/393e31f49ea041eba473c7c266f1a95e
I20260620 23:28:29.375793 2742 ts_tablet_manager.cc:1939] T 393e31f49ea041eba473c7c266f1a95e P 4186cebe1f2e45a08fae117df36fbc02: Deleting consensus metadata
I20260620 23:28:29.376219 2740 ts_tablet_manager.cc:1918] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:29.376477 2740 log.cc:1196] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/393e31f49ea041eba473c7c266f1a95e
I20260620 23:28:29.376938 2129 catalog_manager.cc:4841] TS 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395): tablet 393e31f49ea041eba473c7c266f1a95e (table test-table [id=66df09f4149b432c901bdea525acd61c]) successfully deleted
I20260620 23:28:29.377099 2740 ts_tablet_manager.cc:1939] T 393e31f49ea041eba473c7c266f1a95e P e4c9c36610194b32810b5eb1724258fd: Deleting consensus metadata
I20260620 23:28:29.378549 2129 catalog_manager.cc:4841] TS e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313): tablet 393e31f49ea041eba473c7c266f1a95e (table test-table [id=66df09f4149b432c901bdea525acd61c]) successfully deleted
I20260620 23:28:29.381429 2741 ts_tablet_manager.cc:1918] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:28:29.381620 2741 log.cc:1196] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/393e31f49ea041eba473c7c266f1a95e
I20260620 23:28:29.382092 2741 ts_tablet_manager.cc:1939] T 393e31f49ea041eba473c7c266f1a95e P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting consensus metadata
I20260620 23:28:29.383713 2419 tablet_service.cc:1467] Processing CreateTablet for tablet d1f263f3ad70407bb867c23f87181b3d (DEFAULT_TABLE table=test-table [id=9e54f37242504d5baa6b19b53cce4fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:29.384136 2419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f263f3ad70407bb867c23f87181b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:29.384213 2560 tablet_service.cc:1467] Processing CreateTablet for tablet d1f263f3ad70407bb867c23f87181b3d (DEFAULT_TABLE table=test-table [id=9e54f37242504d5baa6b19b53cce4fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:29.384795 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f263f3ad70407bb867c23f87181b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:29.389035 2751 tablet_bootstrap.cc:492] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Bootstrap starting.
I20260620 23:28:29.390704 2751 tablet_bootstrap.cc:654] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:29.392494 2130 catalog_manager.cc:4841] TS df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194:34037): tablet 393e31f49ea041eba473c7c266f1a95e (table test-table [id=66df09f4149b432c901bdea525acd61c]) successfully deleted
I20260620 23:28:29.393584 2751 tablet_bootstrap.cc:492] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: No bootstrap required, opened a new log
I20260620 23:28:29.393821 2751 ts_tablet_manager.cc:1397] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260620 23:28:29.393805 2279 tablet_service.cc:1467] Processing CreateTablet for tablet d1f263f3ad70407bb867c23f87181b3d (DEFAULT_TABLE table=test-table [id=9e54f37242504d5baa6b19b53cce4fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:29.394228 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f263f3ad70407bb867c23f87181b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:29.394621 2751 raft_consensus.cc:348] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.394891 2751 raft_consensus.cc:374] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:29.394990 2751 raft_consensus.cc:729] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Initialized, Role: FOLLOWER
I20260620 23:28:29.395298 2751 consensus_queue.cc:260] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.395901 2751 ts_tablet_manager.cc:1428] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:28:29.399966 2752 tablet_bootstrap.cc:492] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: Bootstrap starting.
I20260620 23:28:29.400936 2756 tablet_bootstrap.cc:492] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Bootstrap starting.
I20260620 23:28:29.401712 2752 tablet_bootstrap.cc:654] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:29.403051 2756 tablet_bootstrap.cc:654] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:29.404261 2752 tablet_bootstrap.cc:492] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: No bootstrap required, opened a new log
I20260620 23:28:29.404620 2752 ts_tablet_manager.cc:1397] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 23:28:29.405652 2752 raft_consensus.cc:348] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.405901 2752 raft_consensus.cc:374] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:29.405992 2752 raft_consensus.cc:729] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4186cebe1f2e45a08fae117df36fbc02, State: Initialized, Role: FOLLOWER
I20260620 23:28:29.406276 2752 consensus_queue.cc:260] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.406900 2752 ts_tablet_manager.cc:1428] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 23:28:29.407552 2756 tablet_bootstrap.cc:492] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: No bootstrap required, opened a new log
I20260620 23:28:29.407760 2756 ts_tablet_manager.cc:1397] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 23:28:29.408562 2756 raft_consensus.cc:348] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.408841 2756 raft_consensus.cc:374] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:29.408937 2756 raft_consensus.cc:729] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c9c36610194b32810b5eb1724258fd, State: Initialized, Role: FOLLOWER
I20260620 23:28:29.409255 2756 consensus_queue.cc:260] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.409914 2756 ts_tablet_manager.cc:1428] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:28:29.495066 2758 raft_consensus.cc:482] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:29.495383 2758 raft_consensus.cc:504] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.496233 2758 leader_election.cc:288] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:29.496812 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f263f3ad70407bb867c23f87181b3d" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd" is_pre_election: true
I20260620 23:28:29.496897 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f263f3ad70407bb867c23f87181b3d" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02" is_pre_election: true
I20260620 23:28:29.497098 2585 raft_consensus.cc:2454] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 0.
I20260620 23:28:29.497291 2300 raft_consensus.cc:2454] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 0.
I20260620 23:28:29.497507 2372 leader_election.cc:302] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df90a9c4b5714ae4a6d4dc1ec232749c, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:29.497831 2758 raft_consensus.cc:2781] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:29.497948 2758 raft_consensus.cc:482] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:29.498076 2758 raft_consensus.cc:3037] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:29.500284 2758 raft_consensus.cc:504] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.501106 2758 leader_election.cc:288] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 election: Requested vote from peers 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395), e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313)
I20260620 23:28:29.501523 2585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f263f3ad70407bb867c23f87181b3d" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c9c36610194b32810b5eb1724258fd"
I20260620 23:28:29.501560 2300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f263f3ad70407bb867c23f87181b3d" candidate_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4186cebe1f2e45a08fae117df36fbc02"
I20260620 23:28:29.501775 2300 raft_consensus.cc:3037] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:29.501766 2585 raft_consensus.cc:3037] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:29.504865 2300 raft_consensus.cc:2454] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 1.
I20260620 23:28:29.504853 2585 raft_consensus.cc:2454] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df90a9c4b5714ae4a6d4dc1ec232749c in term 1.
I20260620 23:28:29.505479 2372 leader_election.cc:302] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df90a9c4b5714ae4a6d4dc1ec232749c, e4c9c36610194b32810b5eb1724258fd; no voters:
I20260620 23:28:29.505764 2758 raft_consensus.cc:2781] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:29.506247 2758 raft_consensus.cc:686] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 LEADER]: Becoming Leader. State: Replica: df90a9c4b5714ae4a6d4dc1ec232749c, State: Running, Role: LEADER
I20260620 23:28:29.506505 2758 consensus_queue.cc:237] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } }
I20260620 23:28:29.509472 2141 catalog_manager.cc:5495] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c reported cstate change: term changed from 0 to 1, leader changed from <none> to df90a9c4b5714ae4a6d4dc1ec232749c (127.0.96.194). New cstate: current_term: 1 leader_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df90a9c4b5714ae4a6d4dc1ec232749c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 } health_report { overall_health: UNKNOWN } } }
I20260620 23:28:29.543830 2300 raft_consensus.cc:1261] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Refusing update from remote peer df90a9c4b5714ae4a6d4dc1ec232749c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:29.544659 2584 raft_consensus.cc:1261] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Refusing update from remote peer df90a9c4b5714ae4a6d4dc1ec232749c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:29.545459 2755 consensus_queue.cc:1035] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4186cebe1f2e45a08fae117df36fbc02" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:29.545989 2768 consensus_queue.cc:1035] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c9c36610194b32810b5eb1724258fd" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260620 23:28:29.578652 2723 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 23:28:31.181941 2141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:31.182241 2141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:31.187757 2141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98f354658d324f6ab4830c0cbb9f68e0: Sending DeleteTablet for 3 replicas of tablet d1f263f3ad70407bb867c23f87181b3d
I20260620 23:28:31.188727 2419 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f263f3ad70407bb867c23f87181b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:31 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:60638
I20260620 23:28:31.188925 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f263f3ad70407bb867c23f87181b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:31 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42636
I20260620 23:28:31.188874 2279 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f263f3ad70407bb867c23f87181b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:31 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:56316
I20260620 23:28:31.191882 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2214
I20260620 23:28:31.201668 2801 tablet_replica.cc:330] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: stopping tablet replica
I20260620 23:28:31.202335 2801 raft_consensus.cc:2229] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:31.202888 2801 raft_consensus.cc:2258] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:31.205145 2801 ts_tablet_manager.cc:1905] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:31.208000 2799 tablet_replica.cc:330] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: stopping tablet replica
I20260620 23:28:31.208261 2799 raft_consensus.cc:2229] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:31.208627 2799 raft_consensus.cc:2258] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:31.208868 2800 tablet_replica.cc:330] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: stopping tablet replica
I20260620 23:28:31.209729 2799 ts_tablet_manager.cc:1905] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:31.211695 2800 raft_consensus.cc:2229] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:31.212147 2800 raft_consensus.cc:2258] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:31.213485 2800 ts_tablet_manager.cc:1905] T d1f263f3ad70407bb867c23f87181b3d P 4186cebe1f2e45a08fae117df36fbc02: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:28:31 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:31.230406 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.193577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.0.96.193:43395)
0620 23:28:31.206972 (+ 13395us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:31.206982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:31.207925 (+ 943us) server_negotiation.cc:407] Connection header received
0620 23:28:31.208012 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:31.208020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:31.208124 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:31.208252 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:31.210544 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:31.211829 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:31.213946 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:31.214402 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:31.214539 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:31.223731 (+ 9192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:31.223749 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:31.223769 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:31.223818 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:31.226907 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:31.227974 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:31.227983 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:31.227985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:31.228152 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:31.228557 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:31.228561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:31.228563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:31.228950 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:31.229355 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:31.229780 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 23:28:31.230019 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13215,"thread_start_us":12303,"threads_started":1}
W20260620 23:28:31.231295 2345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43808
W20260620 23:28:31.232187 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:31.232443 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 2214
I20260620 23:28:31.315027 2801 ts_tablet_manager.cc:1918] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:28:31.315068 2799 ts_tablet_manager.cc:1918] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 23:28:31.315196 2799 log.cc:1196] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/d1f263f3ad70407bb867c23f87181b3d
I20260620 23:28:31.315426 2801 log.cc:1196] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/d1f263f3ad70407bb867c23f87181b3d
I20260620 23:28:31.315723 2799 ts_tablet_manager.cc:1939] T d1f263f3ad70407bb867c23f87181b3d P df90a9c4b5714ae4a6d4dc1ec232749c: Deleting consensus metadata
I20260620 23:28:31.315972 2801 ts_tablet_manager.cc:1939] T d1f263f3ad70407bb867c23f87181b3d P e4c9c36610194b32810b5eb1724258fd: Deleting consensus metadata
W20260620 23:28:31.341799 2129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:43395 (error 108)
I20260620 23:28:31.342054 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2354
I20260620 23:28:31.344730 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.343670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.0.1:43816)
0620 23:28:31.344351 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.344507 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":426,"threads_started":1}
W20260620 23:28:31.345023 2129 catalog_manager.cc:4584] TS 4186cebe1f2e45a08fae117df36fbc02 (127.0.96.193:43395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1f263f3ad70407bb867c23f87181b3d: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43395: connect: Connection refused (error 111)
Jun 20 23:28:31 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:31.361145 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.342985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.0.96.194:34037)
0620 23:28:31.343559 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:31.343564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:31.343580 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:28:31.343645 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:31.343649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:31.343715 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:31.343853 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:31.345480 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:31.346336 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:31.347667 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:31.348133 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:31.348277 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:31.353378 (+ 5101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:31.353403 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:31.353425 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:31.353483 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:31.357540 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:31.358534 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:31.358544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:31.358547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:31.358741 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:31.359165 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:31.359172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:31.359176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:31.359630 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:31.359982 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:31.360333 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 23:28:31.360665 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":270,"threads_started":1}
W20260620 23:28:31.362113 2485 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36796
W20260620 23:28:31.362919 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:31.363091 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 2354
I20260620 23:28:31.386658 2129 catalog_manager.cc:4841] TS e4c9c36610194b32810b5eb1724258fd (127.0.96.195:37313): tablet d1f263f3ad70407bb867c23f87181b3d (table test-table [id=9e54f37242504d5baa6b19b53cce4fe8]) successfully deleted
I20260620 23:28:31.399500 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.398927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.0.1:43818)
0620 23:28:31.399177 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.399288 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 23:28:31.400655 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.400253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.0.1:43822)
0620 23:28:31.400410 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.400498 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:28:31.407786 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2494
I20260620 23:28:31.408555 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.408044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.0.1:36808)
0620 23:28:31.408227 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.408361 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.194:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Jun 20 23:28:31 dist-test-slave-v34r krb5kdc[2098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998108, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:31.426883 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.408728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58084 (local address 127.0.96.195:37313)
0620 23:28:31.409301 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:31.409306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:31.409325 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:28:31.409434 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:31.409439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:31.409507 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:31.409606 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:31.411164 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:31.411920 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:31.413344 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:31.413794 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:31.413911 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:31.420430 (+ 6519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:31.420454 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:31.420477 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:31.420534 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:31.423731 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:31.424594 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:31.424602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:31.424605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:31.424748 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:31.425195 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:31.425199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:31.425201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:31.425669 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:31.425937 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:31.426294 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 23:28:31.426557 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":340,"threads_started":1}
W20260620 23:28:31.427793 2625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58084
W20260620 23:28:31.428965 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:31.429374 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 2494
I20260620 23:28:31.441843 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.441154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.0.1:36824)
0620 23:28:31.441515 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.441635 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.194:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 23:28:31.443171 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.442826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34037 (local address 127.0.0.1:36828)
0620 23:28:31.442994 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.443068 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.194:34037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:28:31.457224 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2111
I20260620 23:28:31.458716 2183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:31.596714 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:31.595601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43395 (local address 127.0.0.1:43828)
0620 23:28:31.596185 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:31.596339 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409}
I20260620 23:28:31.596729 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 2111
2026-06-20T23:28:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9264 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:31 dist-test-slave-v34r krb5kdc[2812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:31 dist-test-slave-v34r krb5kdc[2812](info): set up 2 sockets
Jun 20 23:28:31 dist-test-slave-v34r krb5kdc[2812](info): commencing operation
krb5kdc: starting...
W20260620 23:28:32.713403 387 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:32 dist-test-slave-v34r krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:32.933595 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34097
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:33.274226 2825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:33.274570 2825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:33.274720 2825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:33.282176 2825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:33.282368 2825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:33.282469 2825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:33.282564 2825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:33.282642 2825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:33.292595 2825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:33.293927 2825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:33.295210 2825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:33.308249 2830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:33.308255 2833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:33.308158 2831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:33.309644 2825 server_base.cc:1034] running on GCE node
I20260620 23:28:33.310428 2825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:33.311847 2825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:33.313097 2825 hybrid_clock.cc:648] HybridClock initialized: now 1781998113313048 us; error 48 us; skew 500 ppm
I20260620 23:28:33.313462 2825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:33 dist-test-slave-v34r krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:33.316898 2825 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:33.318614 2825 webserver.cc:460] Webserver started at http://127.0.96.254:41647/ using document root <none> and password file <none>
I20260620 23:28:33.319335 2825 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:33.319444 2825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:33.319809 2825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:33.323693 2825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "6ba4aff3003744d78c3ca67146b9528c"
format_stamp: "Formatted at 2026-06-20 23:28:33 on dist-test-slave-v34r"
server_key: "a14146f29e87f47ffaaed03003aebd9f"
server_key_iv: "c2d4cccffa67f21da7a38b41aaf1cde2"
server_key_version: "encryptionkey@0"
I20260620 23:28:33.324575 2825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "6ba4aff3003744d78c3ca67146b9528c"
format_stamp: "Formatted at 2026-06-20 23:28:33 on dist-test-slave-v34r"
server_key: "a14146f29e87f47ffaaed03003aebd9f"
server_key_iv: "c2d4cccffa67f21da7a38b41aaf1cde2"
server_key_version: "encryptionkey@0"
I20260620 23:28:33.330240 2825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 23:28:33.334174 2840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:33.335820 2825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:28:33.336035 2825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "6ba4aff3003744d78c3ca67146b9528c"
format_stamp: "Formatted at 2026-06-20 23:28:33 on dist-test-slave-v34r"
server_key: "a14146f29e87f47ffaaed03003aebd9f"
server_key_iv: "c2d4cccffa67f21da7a38b41aaf1cde2"
server_key_version: "encryptionkey@0"
I20260620 23:28:33.336236 2825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:33.355427 2825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:33.359920 2825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:33.360221 2825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:33.381608 2825 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34097
I20260620 23:28:33.381712 2891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34097 every 8 connection(s)
I20260620 23:28:33.383060 2825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:33.384837 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 2825
I20260620 23:28:33.385133 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:33.386000 387 external_mini_cluster.cc:1428] Setting key 8b6b6cd8b4adde55d084fa1a298497b5
I20260620 23:28:33.389242 2892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:28:33 dist-test-slave-v34r krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:33.403294 2892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c: Bootstrap starting.
I20260620 23:28:33.408105 2892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:33.409879 2892 log.cc:824] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:33.411056 2895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:33.388631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.96.254:34097)
0620 23:28:33.389713 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:33.389728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:33.389781 (+ 53us) server_negotiation.cc:407] Connection header received
0620 23:28:33.391467 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:33.391490 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:33.391928 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:33.392366 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:33.393952 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:33.395450 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:33.396738 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:33.397511 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:33.397729 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:33.402464 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:33.402473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:33.402491 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:33.402532 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:33.405954 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:33.407113 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:33.407119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:33.407127 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:33.407302 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:33.407766 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:33.407769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:33.407771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:33.408396 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:33.408656 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:33.409259 (+ 603us) server_negotiation.cc:299] Negotiation successful
0620 23:28:33.409682 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":514,"threads_started":1}
I20260620 23:28:33.413671 2892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c: No bootstrap required, opened a new log
I20260620 23:28:33.422365 2892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba4aff3003744d78c3ca67146b9528c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34097 } }
I20260620 23:28:33.422963 2892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:33.423089 2892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba4aff3003744d78c3ca67146b9528c, State: Initialized, Role: FOLLOWER
I20260620 23:28:33.423858 2892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba4aff3003744d78c3ca67146b9528c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34097 } }
I20260620 23:28:33.424158 2892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:33.424317 2892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:33.424513 2892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:33.427122 2892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba4aff3003744d78c3ca67146b9528c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34097 } }
I20260620 23:28:33.427912 2892 leader_election.cc:302] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ba4aff3003744d78c3ca67146b9528c; no voters:
I20260620 23:28:33.428642 2892 leader_election.cc:288] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:33.428853 2897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:33.429725 2897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [term 1 LEADER]: Becoming Leader. State: Replica: 6ba4aff3003744d78c3ca67146b9528c, State: Running, Role: LEADER
I20260620 23:28:33.430467 2897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba4aff3003744d78c3ca67146b9528c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34097 } }
I20260620 23:28:33.431243 2892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:33.434805 2898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ba4aff3003744d78c3ca67146b9528c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba4aff3003744d78c3ca67146b9528c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34097 } } }
I20260620 23:28:33.434562 2899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ba4aff3003744d78c3ca67146b9528c. Latest consensus state: current_term: 1 leader_uuid: "6ba4aff3003744d78c3ca67146b9528c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ba4aff3003744d78c3ca67146b9528c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34097 } } }
I20260620 23:28:33.435384 2898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:33.435467 2899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:33.436466 2905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:33.443316 2905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:33.454486 2905 catalog_manager.cc:1347] Generated new cluster ID: e8f52a4b80df4da981978f83384b5e6f
I20260620 23:28:33.454672 2905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:33.473440 2905 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:33.475505 2905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:33.487330 2905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ba4aff3003744d78c3ca67146b9528c: Generated new TSK 0
I20260620 23:28:33.488559 2905 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:33.553159 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34097
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:33.963548 2918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:33.964001 2918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:33.964198 2918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:33.972517 2918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:33.972776 2918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:33.973034 2918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:33.984515 2918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:33.986413 2918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:33.988143 2918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:34.004799 2923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:34.004931 2924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:34.004827 2926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:34.010972 2918 server_base.cc:1034] running on GCE node
I20260620 23:28:34.011962 2918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:34.013368 2918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:34.014668 2918 hybrid_clock.cc:648] HybridClock initialized: now 1781998114014605 us; error 54 us; skew 500 ppm
I20260620 23:28:34.015106 2918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:34 dist-test-slave-v34r krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:34.019287 2918 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:34.024549 2918 webserver.cc:460] Webserver started at http://127.0.96.193:33025/ using document root <none> and password file <none>
I20260620 23:28:34.025807 2918 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:34.026021 2918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:34.026461 2918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:34.030021 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "6d5e024b4ca34b518069614c608c7beb"
format_stamp: "Formatted at 2026-06-20 23:28:34 on dist-test-slave-v34r"
server_key: "c853fc31b11fc9de3379fae147a7d0a8"
server_key_iv: "cc0dc5798e85a6030a965083eac3ca34"
server_key_version: "encryptionkey@0"
I20260620 23:28:34.030901 2918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "6d5e024b4ca34b518069614c608c7beb"
format_stamp: "Formatted at 2026-06-20 23:28:34 on dist-test-slave-v34r"
server_key: "c853fc31b11fc9de3379fae147a7d0a8"
server_key_iv: "cc0dc5798e85a6030a965083eac3ca34"
server_key_version: "encryptionkey@0"
I20260620 23:28:34.037243 2918 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 23:28:34.041785 2933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:34.043656 2918 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:28:34.043877 2918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "6d5e024b4ca34b518069614c608c7beb"
format_stamp: "Formatted at 2026-06-20 23:28:34 on dist-test-slave-v34r"
server_key: "c853fc31b11fc9de3379fae147a7d0a8"
server_key_iv: "cc0dc5798e85a6030a965083eac3ca34"
server_key_version: "encryptionkey@0"
I20260620 23:28:34.044097 2918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:34.063094 2918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:34.067782 2918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:34.068101 2918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:34.069304 2918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:34.070911 2918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:34.071022 2918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:34.071184 2918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:34.071259 2918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:34.110620 2918 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:44729
I20260620 23:28:34.110724 3045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:44729 every 8 connection(s)
I20260620 23:28:34.111989 2918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:34.118613 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 2918
I20260620 23:28:34.118898 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:34.119616 387 external_mini_cluster.cc:1428] Setting key e279d61b9b35e3f41953d0cb6d8dfa82
Jun 20 23:28:34 dist-test-slave-v34r krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:34.136814 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:34.115610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:43487 (local address 127.0.96.254:34097)
0620 23:28:34.116127 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:34.116131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:34.117591 (+ 1460us) server_negotiation.cc:407] Connection header received
0620 23:28:34.120081 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:34.120086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:34.120186 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:34.120304 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:34.122229 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.122952 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.124555 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.124943 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.125138 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:34.128763 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:34.128769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:34.128772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:34.128802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:34.132587 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:34.133766 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:34.133775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:34.133781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:34.133958 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:34.134477 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:34.134484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:34.134491 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:34.134883 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:34.135044 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:34.136040 (+ 996us) server_negotiation.cc:299] Negotiation successful
0620 23:28:34.136317 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":288,"threads_started":1}
I20260620 23:28:34.138161 3049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:34.116385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34097 (local address 127.0.96.193:43487)
0620 23:28:34.117389 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:34.117445 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:34.119767 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:34.120580 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:34.120590 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:34.121348 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:34.122049 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:34.122067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.123117 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.123121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:34.124399 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:34.124411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.125230 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.125239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:34.125489 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:34.126294 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:34.126322 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:34.128512 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:34.132862 (+ 4350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:34.132874 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:34.132895 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:34.133597 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:34.134095 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:34.134099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:34.134102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:34.134335 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:34.135038 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:34.135048 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:34.135545 (+ 497us) client_negotiation.cc:769] Sending connection context
0620 23:28:34.135851 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 23:28:34.136436 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":356,"threads_started":1}
I20260620 23:28:34.140411 3046 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34097
I20260620 23:28:34.140846 3046 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:28:34.141956 3046 heartbeater.cc:511] Master 127.0.96.254:34097 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:28:34.145532 2857 ts_manager.cc:194] Registered new tserver with Master: 6d5e024b4ca34b518069614c608c7beb (127.0.96.193:44729)
I20260620 23:28:34.148296 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:43487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:34.170399 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34097
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:34.574020 3052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:34.574483 3052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:34.574666 3052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:34.583213 3052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:34.583434 3052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:34.583653 3052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:34.595458 3052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:34.597412 3052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:34.599117 3052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:34.617178 3057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:34.617595 3058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:34.617868 3060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:34.618494 3052 server_base.cc:1034] running on GCE node
I20260620 23:28:34.619372 3052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:34.620621 3052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:34.621903 3052 hybrid_clock.cc:648] HybridClock initialized: now 1781998114621816 us; error 82 us; skew 500 ppm
I20260620 23:28:34.622335 3052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:34 dist-test-slave-v34r krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:34.626439 3052 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:34.628695 3052 webserver.cc:460] Webserver started at http://127.0.96.194:41971/ using document root <none> and password file <none>
I20260620 23:28:34.629792 3052 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:34.629987 3052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:34.630406 3052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:34.634876 3052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "9176cdf2abd8434eb50d7120f249cb88"
format_stamp: "Formatted at 2026-06-20 23:28:34 on dist-test-slave-v34r"
server_key: "8b68eb1e3978f9630e0419f172cdde27"
server_key_iv: "44dc2c1946ba2dccc2b7f009544682e4"
server_key_version: "encryptionkey@0"
I20260620 23:28:34.635994 3052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "9176cdf2abd8434eb50d7120f249cb88"
format_stamp: "Formatted at 2026-06-20 23:28:34 on dist-test-slave-v34r"
server_key: "8b68eb1e3978f9630e0419f172cdde27"
server_key_iv: "44dc2c1946ba2dccc2b7f009544682e4"
server_key_version: "encryptionkey@0"
I20260620 23:28:34.643657 3052 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 23:28:34.648900 3067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:34.651160 3052 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:28:34.651433 3052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "9176cdf2abd8434eb50d7120f249cb88"
format_stamp: "Formatted at 2026-06-20 23:28:34 on dist-test-slave-v34r"
server_key: "8b68eb1e3978f9630e0419f172cdde27"
server_key_iv: "44dc2c1946ba2dccc2b7f009544682e4"
server_key_version: "encryptionkey@0"
I20260620 23:28:34.651728 3052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:34.666399 3052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:34.671242 3052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:34.671557 3052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:34.672608 3052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:34.674127 3052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:34.674247 3052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:34.674417 3052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:34.674489 3052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:34.712077 3052 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:40077
I20260620 23:28:34.712210 3179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:40077 every 8 connection(s)
I20260620 23:28:34.713984 3052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:34.724421 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3052
I20260620 23:28:34.724686 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:34.725329 387 external_mini_cluster.cc:1428] Setting key a142c1341352d349242e33db58e7f40d
Jun 20 23:28:34 dist-test-slave-v34r krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:34.740217 3182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:34.717533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:56657 (local address 127.0.96.254:34097)
0620 23:28:34.718128 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:34.718133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:34.719512 (+ 1379us) server_negotiation.cc:407] Connection header received
0620 23:28:34.721631 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:34.721637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:34.721738 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:34.721863 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:34.723565 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.724313 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.726237 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.726831 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.726985 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:34.731857 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:34.731868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:34.731876 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:34.731933 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:34.735952 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:34.737091 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:34.737099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:34.737105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:34.737274 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:34.737767 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:34.737775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:34.737781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:34.738217 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:34.738386 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:34.739420 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0620 23:28:34.739691 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":474,"thread_start_us":317,"threads_started":1}
I20260620 23:28:34.741674 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:34.718330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34097 (local address 127.0.96.194:56657)
0620 23:28:34.719289 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:34.719346 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:34.721305 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:34.722143 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:34.722153 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:34.722732 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:34.723371 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:34.723391 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.724480 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.724484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:34.726078 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:34.726092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:34.727184 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:34.727191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:34.727621 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:34.728540 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:34.728575 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:34.731484 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:34.736188 (+ 4704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:34.736199 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:34.736221 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:34.736834 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:34.737396 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:34.737399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:34.737402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:34.737624 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:34.738393 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:34.738401 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:34.738896 (+ 495us) client_negotiation.cc:769] Sending connection context
0620 23:28:34.739262 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 23:28:34.739888 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":374,"threads_started":1}
I20260620 23:28:34.743670 3180 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34097
I20260620 23:28:34.744087 3180 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:34.745124 3180 heartbeater.cc:511] Master 127.0.96.254:34097 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:28:34.747120 2857 ts_manager.cc:194] Registered new tserver with Master: 9176cdf2abd8434eb50d7120f249cb88 (127.0.96.194:40077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:28:34.748413 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:56657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:34.773075 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34097
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:35.124605 3186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:35.125046 3186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:35.125242 3186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:35.133795 3186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:35.133947 3186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:35.134177 3186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:35.146301 3186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:35.148204 3186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:35.150032 3186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:28:35.152478 3046 heartbeater.cc:503] Master 127.0.96.254:34097 was elected leader, sending a full tablet report...
W20260620 23:28:35.167033 3192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:35.167078 3194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:35.167150 3191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:35.171125 3186 server_base.cc:1034] running on GCE node
I20260620 23:28:35.171980 3186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:35.173245 3186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:35.174505 3186 hybrid_clock.cc:648] HybridClock initialized: now 1781998115174413 us; error 90 us; skew 500 ppm
I20260620 23:28:35.174980 3186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:35 dist-test-slave-v34r krb5kdc[2812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:35.179369 3186 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:35.181500 3186 webserver.cc:460] Webserver started at http://127.0.96.195:40395/ using document root <none> and password file <none>
I20260620 23:28:35.182483 3186 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:35.182664 3186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:35.183120 3186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:35.186895 3186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "5ec7052080684401bac292502b6658ae"
format_stamp: "Formatted at 2026-06-20 23:28:35 on dist-test-slave-v34r"
server_key: "b46facbf601f931577fd9ddcd3432f24"
server_key_iv: "f6220afeb54c8923d1b83e38761d767c"
server_key_version: "encryptionkey@0"
I20260620 23:28:35.187904 3186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "5ec7052080684401bac292502b6658ae"
format_stamp: "Formatted at 2026-06-20 23:28:35 on dist-test-slave-v34r"
server_key: "b46facbf601f931577fd9ddcd3432f24"
server_key_iv: "f6220afeb54c8923d1b83e38761d767c"
server_key_version: "encryptionkey@0"
I20260620 23:28:35.194685 3186 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 23:28:35.199379 3201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:35.201373 3186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:28:35.201598 3186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "5ec7052080684401bac292502b6658ae"
format_stamp: "Formatted at 2026-06-20 23:28:35 on dist-test-slave-v34r"
server_key: "b46facbf601f931577fd9ddcd3432f24"
server_key_iv: "f6220afeb54c8923d1b83e38761d767c"
server_key_version: "encryptionkey@0"
I20260620 23:28:35.201896 3186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:35.234372 3186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:35.240084 3186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:35.240437 3186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:35.241762 3186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:35.243638 3186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:35.243762 3186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:35.243925 3186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:35.244009 3186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:35.283982 3186 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:43737
I20260620 23:28:35.284042 3313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:43737 every 8 connection(s)
I20260620 23:28:35.285669 3186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:35.287046 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3186
I20260620 23:28:35.287261 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:35.287838 387 external_mini_cluster.cc:1428] Setting key 9e4586954a35b93f5dd7b7f6f969050e
Jun 20 23:28:35 dist-test-slave-v34r krb5kdc[2812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:35.310300 3316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:35.289178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:35853 (local address 127.0.96.254:34097)
0620 23:28:35.289738 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:35.289744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:35.291551 (+ 1807us) server_negotiation.cc:407] Connection header received
0620 23:28:35.293843 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:35.293848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:35.293937 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:35.294057 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:35.295747 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:35.296609 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:35.298572 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:35.299033 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:35.299163 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:35.303144 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:35.303151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:35.303155 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:35.303204 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:35.307135 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:35.307982 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:35.307987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:35.307989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:35.308134 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:35.308567 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:35.308570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:35.308572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:35.308876 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:35.309079 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:35.309781 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 23:28:35.309984 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":240,"threads_started":1}
I20260620 23:28:35.311275 3317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:35.290109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34097 (local address 127.0.96.195:35853)
0620 23:28:35.291258 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:35.291326 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:35.293519 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:35.294327 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:35.294335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:35.294911 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:35.295545 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:35.295561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:35.296809 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:35.296814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:35.298387 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:35.298401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:35.299275 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:35.299281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:35.299548 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:35.300378 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:35.300404 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:35.302817 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:35.307301 (+ 4484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:35.307311 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:35.307369 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:35.307835 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:35.308236 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:35.308239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:35.308241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:35.308456 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:35.309026 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:35.309034 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:35.309419 (+ 385us) client_negotiation.cc:769] Sending connection context
0620 23:28:35.309657 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 23:28:35.310054 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":541,"threads_started":1}
I20260620 23:28:35.313252 3314 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34097
I20260620 23:28:35.313618 3314 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:35.314411 3314 heartbeater.cc:511] Master 127.0.96.254:34097 requested a full tablet report, sending...
I20260620 23:28:35.316260 2857 ts_manager.cc:194] Registered new tserver with Master: 5ec7052080684401bac292502b6658ae (127.0.96.195:43737)
I20260620 23:28:35.317319 2857 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:35853
I20260620 23:28:35.318724 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:28:35.354521 3316 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:28:35.351618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.96.254:34097)
0620 23:28:35.351826 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:35.351832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:35.352321 (+ 489us) server_negotiation.cc:407] Connection header received
0620 23:28:35.352493 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:35.352498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:35.352582 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:35.352731 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:35.354196 (+ 1465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56696: BlockingRecv error: recv got EOF from 127.0.0.1:56696 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:28:35.357998 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2918
W20260620 23:28:35.361418 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2918: Not authorized: Client connection negotiation failed: client connection to 127.0.96.193:44729: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:28:35.361577 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 2918
I20260620 23:28:35.377925 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3052
W20260620 23:28:35.381028 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3052: Not authorized: Client connection negotiation failed: client connection to 127.0.96.194:40077: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:28:35.381287 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3052
I20260620 23:28:35.399091 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3186
W20260620 23:28:35.402452 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3186: Not authorized: Client connection negotiation failed: client connection to 127.0.96.195:43737: server requires authentication, but client does not have Kerberos credentials available
I20260620 23:28:35.402606 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3186
I20260620 23:28:35.419020 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 2825
I20260620 23:28:35.420740 2887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:35.524158 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 2825
2026-06-20T23:28:35Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3930 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:35 dist-test-slave-v34r krb5kdc[3332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:35 dist-test-slave-v34r krb5kdc[3332](info): set up 2 sockets
Jun 20 23:28:35 dist-test-slave-v34r krb5kdc[3332](info): commencing operation
krb5kdc: starting...
W20260620 23:28:36.645576 387 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:36 dist-test-slave-v34r krb5kdc[3332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:36.872882 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:43547
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:37.285055 3345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:37.285468 3345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:37.285571 3345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:37.293771 3345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:37.293928 3345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:37.293984 3345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:37.294039 3345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:37.294095 3345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:37.305441 3345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:37.307366 3345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:37.309158 3345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:37.323978 3350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:37.323978 3351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:37.324090 3353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:37.330648 3345 server_base.cc:1034] running on GCE node
I20260620 23:28:37.331844 3345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:37.333863 3345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:37.335161 3345 hybrid_clock.cc:648] HybridClock initialized: now 1781998117335111 us; error 53 us; skew 500 ppm
I20260620 23:28:37.335660 3345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:37 dist-test-slave-v34r krb5kdc[3332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:37.352736 3345 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:37.355146 3345 webserver.cc:460] Webserver started at http://127.0.96.254:42283/ using document root <none> and password file <none>
I20260620 23:28:37.356230 3345 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:37.356432 3345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:37.356832 3345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:37.361120 3345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "fba8aaf715c940f688ad13be4b2d2003"
format_stamp: "Formatted at 2026-06-20 23:28:37 on dist-test-slave-v34r"
server_key: "703061406827ffc2b61b768138755301"
server_key_iv: "a934042299adac10f752555271490bc4"
server_key_version: "encryptionkey@0"
I20260620 23:28:37.362215 3345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "fba8aaf715c940f688ad13be4b2d2003"
format_stamp: "Formatted at 2026-06-20 23:28:37 on dist-test-slave-v34r"
server_key: "703061406827ffc2b61b768138755301"
server_key_iv: "a934042299adac10f752555271490bc4"
server_key_version: "encryptionkey@0"
I20260620 23:28:37.369349 3345 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260620 23:28:37.374356 3360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:37.376487 3345 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:37.376819 3345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "fba8aaf715c940f688ad13be4b2d2003"
format_stamp: "Formatted at 2026-06-20 23:28:37 on dist-test-slave-v34r"
server_key: "703061406827ffc2b61b768138755301"
server_key_iv: "a934042299adac10f752555271490bc4"
server_key_version: "encryptionkey@0"
I20260620 23:28:37.377213 3345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:37.392503 3345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:37.397922 3345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:37.398293 3345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:37.421634 3345 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:43547
I20260620 23:28:37.421687 3411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:43547 every 8 connection(s)
I20260620 23:28:37.423492 3345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:37.428216 3412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:37.429768 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3345
I20260620 23:28:37.430015 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:37.430747 387 external_mini_cluster.cc:1428] Setting key 5a1a4b6a420dd5e89c315cab125f792b
I20260620 23:28:37.442787 3412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003: Bootstrap starting.
I20260620 23:28:37.446677 3412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:37.447710 3412 log.cc:824] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:28:37 dist-test-slave-v34r krb5kdc[3332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:37.451346 3412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003: No bootstrap required, opened a new log
I20260620 23:28:37.460561 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:37.433506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36996 (local address 127.0.96.254:43547)
0620 23:28:37.434625 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:37.434646 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:37.434721 (+ 75us) server_negotiation.cc:407] Connection header received
0620 23:28:37.437172 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:37.437204 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:37.437764 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:37.438257 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:37.440120 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:37.442142 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:37.443765 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:37.444884 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:37.445144 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:37.450459 (+ 5315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:37.450470 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:37.450493 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:37.450548 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:37.454521 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:37.455567 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:37.455578 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:37.455588 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:37.455840 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:37.456296 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:37.456300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:37.456302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:37.457147 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:37.457431 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:37.458281 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 23:28:37.458965 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":482,"threads_started":1}
I20260620 23:28:37.461517 3412 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba8aaf715c940f688ad13be4b2d2003" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 43547 } }
I20260620 23:28:37.462018 3412 raft_consensus.cc:374] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:37.462170 3412 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba8aaf715c940f688ad13be4b2d2003, State: Initialized, Role: FOLLOWER
I20260620 23:28:37.462957 3412 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba8aaf715c940f688ad13be4b2d2003" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 43547 } }
I20260620 23:28:37.463261 3412 raft_consensus.cc:388] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:37.463428 3412 raft_consensus.cc:482] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:37.463634 3412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:37.466315 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:28:37.466681 3412 raft_consensus.cc:504] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba8aaf715c940f688ad13be4b2d2003" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 43547 } }
I20260620 23:28:37.467274 3412 leader_election.cc:302] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fba8aaf715c940f688ad13be4b2d2003; no voters:
I20260620 23:28:37.468029 3412 leader_election.cc:288] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:37.468281 3417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:37.469693 3417 raft_consensus.cc:686] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [term 1 LEADER]: Becoming Leader. State: Replica: fba8aaf715c940f688ad13be4b2d2003, State: Running, Role: LEADER
I20260620 23:28:37.470654 3412 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:37.472195 3417 consensus_queue.cc:237] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba8aaf715c940f688ad13be4b2d2003" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 43547 } }
I20260620 23:28:37.476337 3418 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fba8aaf715c940f688ad13be4b2d2003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba8aaf715c940f688ad13be4b2d2003" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 43547 } } }
I20260620 23:28:37.476886 3418 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:37.477563 3419 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fba8aaf715c940f688ad13be4b2d2003. Latest consensus state: current_term: 1 leader_uuid: "fba8aaf715c940f688ad13be4b2d2003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fba8aaf715c940f688ad13be4b2d2003" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 43547 } } }
I20260620 23:28:37.477901 3419 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:37.482008 3428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:37.485123 3435 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:28:37.489130 3428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:37.501214 3428 catalog_manager.cc:1347] Generated new cluster ID: 66afb9b5758c41e18727eb3ec574a8d1
I20260620 23:28:37.501402 3428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:37.531332 3428 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:37.533607 3428 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:37.534695 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:37.523107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37008 (local address 127.0.96.254:43547)
0620 23:28:37.523377 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:37.523386 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:37.524405 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 23:28:37.524653 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:37.524661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:37.524771 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:37.524924 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:37.526441 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:37.527672 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:37.529932 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:37.530725 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:37.530911 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:37.532326 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:37.532334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:37.532340 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:37.533442 (+ 1102us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:37.533545 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:37.533607 (+ 62us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:37.533858 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:37.534071 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 23:28:37.534237 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 23:28:37.546271 3428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fba8aaf715c940f688ad13be4b2d2003: Generated new TSK 0
I20260620 23:28:37.547832 3428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:28:37.558882 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3345
I20260620 23:28:37.560762 3407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:37.666603 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3345
2026-06-20T23:28:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2133 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:37 dist-test-slave-v34r krb5kdc[3446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:37 dist-test-slave-v34r krb5kdc[3446](info): set up 2 sockets
Jun 20 23:28:37 dist-test-slave-v34r krb5kdc[3446](info): commencing operation
krb5kdc: starting...
W20260620 23:28:38.766722 387 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:38 dist-test-slave-v34r krb5kdc[3446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:39.000562 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38383
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:39.416329 3459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:39.416749 3459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:39.416945 3459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:39.425088 3459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:39.425246 3459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:39.425347 3459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:39.425449 3459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:39.425525 3459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:39.436924 3459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:39.438813 3459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:39.440502 3459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:39.454668 3465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:39.454586 3464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:39.454612 3467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:39.455999 3459 server_base.cc:1034] running on GCE node
I20260620 23:28:39.457085 3459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:39.458689 3459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:39.459959 3459 hybrid_clock.cc:648] HybridClock initialized: now 1781998119459911 us; error 53 us; skew 500 ppm
I20260620 23:28:39.460302 3459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:39 dist-test-slave-v34r krb5kdc[3446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:39.463737 3459 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:39.465706 3459 webserver.cc:460] Webserver started at http://127.0.96.254:38081/ using document root <none> and password file <none>
I20260620 23:28:39.466576 3459 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:39.466692 3459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:39.466992 3459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:39.470676 3459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "0eee2cad4d934c61b30cde7db4428789"
format_stamp: "Formatted at 2026-06-20 23:28:39 on dist-test-slave-v34r"
server_key: "28aec45a77c5ab0900184ac23459256c"
server_key_iv: "603665068219663bb964768e3cc19621"
server_key_version: "encryptionkey@0"
I20260620 23:28:39.471554 3459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "0eee2cad4d934c61b30cde7db4428789"
format_stamp: "Formatted at 2026-06-20 23:28:39 on dist-test-slave-v34r"
server_key: "28aec45a77c5ab0900184ac23459256c"
server_key_iv: "603665068219663bb964768e3cc19621"
server_key_version: "encryptionkey@0"
I20260620 23:28:39.478379 3459 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 23:28:39.483073 3474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:39.485148 3459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:28:39.485400 3459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "0eee2cad4d934c61b30cde7db4428789"
format_stamp: "Formatted at 2026-06-20 23:28:39 on dist-test-slave-v34r"
server_key: "28aec45a77c5ab0900184ac23459256c"
server_key_iv: "603665068219663bb964768e3cc19621"
server_key_version: "encryptionkey@0"
I20260620 23:28:39.485643 3459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:39.503736 3459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:39.509518 3459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:39.509900 3459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:39.532393 3459 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38383
I20260620 23:28:39.532439 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38383 every 8 connection(s)
I20260620 23:28:39.534220 3459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:39.535156 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3459
I20260620 23:28:39.535399 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:39.536235 387 external_mini_cluster.cc:1428] Setting key 0284ee705def81232a3260e81e730f46
I20260620 23:28:39.539748 3526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:39.551271 3526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789: Bootstrap starting.
Jun 20 23:28:39 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:39.554762 3526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:39.556097 3526 log.cc:824] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:39.559827 3526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789: No bootstrap required, opened a new log
I20260620 23:28:39.563901 3529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:39.538798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41518 (local address 127.0.96.254:38383)
0620 23:28:39.539770 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:39.539787 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:39.539843 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:28:39.541916 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:39.541949 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:39.542534 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:39.543083 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:39.544817 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:39.546389 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:39.547878 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:39.548918 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:39.549227 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:39.553912 (+ 4685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:39.553923 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:39.553946 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:39.554002 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:39.558368 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:39.559334 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:39.559342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:39.559351 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:39.559559 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:39.560026 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:39.560032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:39.560036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:39.560749 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:39.561211 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:39.561706 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 23:28:39.562298 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":407,"threads_started":1}
I20260620 23:28:39.568871 3526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eee2cad4d934c61b30cde7db4428789" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38383 } }
I20260620 23:28:39.569356 3526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:39.569463 3526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eee2cad4d934c61b30cde7db4428789, State: Initialized, Role: FOLLOWER
I20260620 23:28:39.570055 3526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eee2cad4d934c61b30cde7db4428789" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38383 } }
I20260620 23:28:39.570300 3526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:39.570454 3526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:39.570624 3526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:39.572809 3526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eee2cad4d934c61b30cde7db4428789" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38383 } }
I20260620 23:28:39.573433 3526 leader_election.cc:302] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eee2cad4d934c61b30cde7db4428789; no voters:
I20260620 23:28:39.574101 3526 leader_election.cc:288] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:39.574304 3531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:39.575173 3531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [term 1 LEADER]: Becoming Leader. State: Replica: 0eee2cad4d934c61b30cde7db4428789, State: Running, Role: LEADER
I20260620 23:28:39.575922 3531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eee2cad4d934c61b30cde7db4428789" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38383 } }
I20260620 23:28:39.576398 3526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:39.580488 3532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eee2cad4d934c61b30cde7db4428789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eee2cad4d934c61b30cde7db4428789" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38383 } } }
I20260620 23:28:39.581008 3532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:39.580621 3533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eee2cad4d934c61b30cde7db4428789. Latest consensus state: current_term: 1 leader_uuid: "0eee2cad4d934c61b30cde7db4428789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eee2cad4d934c61b30cde7db4428789" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38383 } } }
I20260620 23:28:39.581154 3533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:39.582105 3538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:39.589327 3538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:39.600333 3538 catalog_manager.cc:1347] Generated new cluster ID: ac813a632e5e4d1a801c5fcb61a2aa2d
I20260620 23:28:39.600500 3538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:39.611177 3538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:39.612844 3538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:39.620432 3538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eee2cad4d934c61b30cde7db4428789: Generated new TSK 0
I20260620 23:28:39.621876 3538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:39.680572 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38383
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:40.057461 3552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:40.057801 3552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:40.057994 3552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:40.066253 3552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:40.066447 3552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:40.066655 3552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:40.077359 3552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:40.078821 3552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:40.080308 3552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:40.096498 3557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:40.096647 3558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:40.096539 3560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:40.097461 3552 server_base.cc:1034] running on GCE node
I20260620 23:28:40.098191 3552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:40.099296 3552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:40.100559 3552 hybrid_clock.cc:648] HybridClock initialized: now 1781998120100482 us; error 61 us; skew 500 ppm
I20260620 23:28:40.101079 3552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:40 dist-test-slave-v34r krb5kdc[3446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:40.104841 3552 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:40.106632 3552 webserver.cc:460] Webserver started at http://127.0.96.193:34133/ using document root <none> and password file <none>
I20260620 23:28:40.107406 3552 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:40.107563 3552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:40.107877 3552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:40.111073 3552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "cad5e27ec85a4b77a897b8e255c97b65"
format_stamp: "Formatted at 2026-06-20 23:28:40 on dist-test-slave-v34r"
server_key: "f5270225c9c7a74b69919fddadb75b2f"
server_key_iv: "6463ddd1bbfbfbecdd8032c703570342"
server_key_version: "encryptionkey@0"
I20260620 23:28:40.111963 3552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "cad5e27ec85a4b77a897b8e255c97b65"
format_stamp: "Formatted at 2026-06-20 23:28:40 on dist-test-slave-v34r"
server_key: "f5270225c9c7a74b69919fddadb75b2f"
server_key_iv: "6463ddd1bbfbfbecdd8032c703570342"
server_key_version: "encryptionkey@0"
I20260620 23:28:40.118158 3552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260620 23:28:40.122726 3567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:40.124727 3552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:40.125089 3552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "cad5e27ec85a4b77a897b8e255c97b65"
format_stamp: "Formatted at 2026-06-20 23:28:40 on dist-test-slave-v34r"
server_key: "f5270225c9c7a74b69919fddadb75b2f"
server_key_iv: "6463ddd1bbfbfbecdd8032c703570342"
server_key_version: "encryptionkey@0"
I20260620 23:28:40.125347 3552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:40.146195 3552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:40.151273 3552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:40.151649 3552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:40.152730 3552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:40.154387 3552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:40.154500 3552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:40.154664 3552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:40.154738 3552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:40.193332 3552 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:39733
I20260620 23:28:40.193437 3679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:39733 every 8 connection(s)
I20260620 23:28:40.194716 3552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:40.196345 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3552
I20260620 23:28:40.196561 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:40.197425 387 external_mini_cluster.cc:1428] Setting key df0d280fe3ed8d6143bbb5f7879d7105
Jun 20 23:28:40 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:28:40.221203 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:40.198370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:55119 (local address 127.0.96.254:38383)
0620 23:28:40.198934 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:40.198942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:40.201033 (+ 2091us) server_negotiation.cc:407] Connection header received
0620 23:28:40.203285 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:40.203296 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:40.203422 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:40.203607 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:40.206110 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.207182 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.209267 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.209727 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.209920 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:40.214418 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:40.214424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:40.214428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:40.214470 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:40.217435 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:40.218339 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:40.218350 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:40.218352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:40.218474 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:40.219081 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:40.219086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:40.219090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:40.219415 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:40.219578 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:40.220456 (+ 878us) server_negotiation.cc:299] Negotiation successful
0620 23:28:40.220728 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
I20260620 23:28:40.222548 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:40.199257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38383 (local address 127.0.96.193:55119)
0620 23:28:40.200760 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:40.200825 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:40.202888 (+ 2063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:40.203909 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:40.203921 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:40.204871 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:40.205844 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:40.205869 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.207425 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.207430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:40.208946 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:40.208964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.210062 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.210070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:40.210366 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:40.211229 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:40.211264 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:40.214069 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:40.217629 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:40.217643 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:40.217669 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:40.218189 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:40.218733 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:40.218737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:40.218739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:40.218953 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:40.219559 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:40.219569 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:40.220020 (+ 451us) client_negotiation.cc:769] Sending connection context
0620 23:28:40.220354 (+ 334us) client_negotiation.cc:240] Negotiation successful
0620 23:28:40.221086 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":899,"threads_started":1}
I20260620 23:28:40.224685 3680 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38383
I20260620 23:28:40.225162 3680 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:40.226157 3680 heartbeater.cc:511] Master 127.0.96.254:38383 requested a full tablet report, sending...
I20260620 23:28:40.229158 3491 ts_manager.cc:194] Registered new tserver with Master: cad5e27ec85a4b77a897b8e255c97b65 (127.0.96.193:39733)
I20260620 23:28:40.231606 3491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:55119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:40.249732 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38383
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:40.590394 3686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:40.590682 3686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:40.590776 3686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:40.598445 3686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:40.598568 3686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:40.598682 3686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:40.608851 3686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:40.610131 3686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:40.611377 3686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:40.626616 3692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:40.626587 3691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:40.627255 3694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:40.627619 3686 server_base.cc:1034] running on GCE node
I20260620 23:28:40.628273 3686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:40.629549 3686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:40.630944 3686 hybrid_clock.cc:648] HybridClock initialized: now 1781998120630850 us; error 75 us; skew 500 ppm
I20260620 23:28:40.631345 3686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:40 dist-test-slave-v34r krb5kdc[3446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:40.635560 3686 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:40.637674 3686 webserver.cc:460] Webserver started at http://127.0.96.194:35381/ using document root <none> and password file <none>
I20260620 23:28:40.638495 3686 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:40.638661 3686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:40.639026 3686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:40.642421 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "b83f9030df0f4c20b56b15888dd27338"
format_stamp: "Formatted at 2026-06-20 23:28:40 on dist-test-slave-v34r"
server_key: "27af247bb0d8ee5e963a7eaab21d25a2"
server_key_iv: "e8fe86fe2b89994a9d6b32037a768654"
server_key_version: "encryptionkey@0"
I20260620 23:28:40.643321 3686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "b83f9030df0f4c20b56b15888dd27338"
format_stamp: "Formatted at 2026-06-20 23:28:40 on dist-test-slave-v34r"
server_key: "27af247bb0d8ee5e963a7eaab21d25a2"
server_key_iv: "e8fe86fe2b89994a9d6b32037a768654"
server_key_version: "encryptionkey@0"
I20260620 23:28:40.650161 3686 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 23:28:40.654359 3701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:40.656131 3686 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:28:40.656371 3686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "b83f9030df0f4c20b56b15888dd27338"
format_stamp: "Formatted at 2026-06-20 23:28:40 on dist-test-slave-v34r"
server_key: "27af247bb0d8ee5e963a7eaab21d25a2"
server_key_iv: "e8fe86fe2b89994a9d6b32037a768654"
server_key_version: "encryptionkey@0"
I20260620 23:28:40.656601 3686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:40.675990 3686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:40.680541 3686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:40.680860 3686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:40.682158 3686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:40.683991 3686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:40.684093 3686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:40.684224 3686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:40.684303 3686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:40.722380 3686 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:46711
I20260620 23:28:40.722481 3813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:46711 every 8 connection(s)
I20260620 23:28:40.723738 3686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:40.731341 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3686
I20260620 23:28:40.731555 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:40.732069 387 external_mini_cluster.cc:1428] Setting key 0d850e519af2c474bc10548098370f88
Jun 20 23:28:40 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:40.747305 3816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:40.727203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:36373 (local address 127.0.96.254:38383)
0620 23:28:40.727772 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:40.727777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:40.729182 (+ 1405us) server_negotiation.cc:407] Connection header received
0620 23:28:40.731274 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:40.731280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:40.731376 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:40.731496 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:40.733353 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.734191 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.735806 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.736253 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.736395 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:40.740684 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:40.740689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:40.740693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:40.740735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:40.743844 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:40.744690 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:40.744696 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:40.744700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:40.744848 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:40.745369 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:40.745373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:40.745375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:40.745669 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:40.745855 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:40.746605 (+ 750us) server_negotiation.cc:299] Negotiation successful
0620 23:28:40.746861 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":288,"threads_started":1}
I20260620 23:28:40.748333 3817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:40.727984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38383 (local address 127.0.96.194:36373)
0620 23:28:40.728919 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:40.729038 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:40.730949 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:40.731883 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:40.731893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:40.732498 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:40.733173 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:40.733194 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.734371 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.734375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:40.735641 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:40.735653 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:40.736495 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:40.736503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:40.736739 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:40.737673 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:40.737701 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:40.740409 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:40.744034 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:40.744045 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:40.744066 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:40.744544 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:40.744971 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:40.744977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:40.745006 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:40.745218 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:40.745830 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:40.745840 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:40.746216 (+ 376us) client_negotiation.cc:769] Sending connection context
0620 23:28:40.746514 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 23:28:40.747106 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":379,"threads_started":1}
I20260620 23:28:40.750181 3814 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38383
I20260620 23:28:40.750702 3814 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:40.751950 3814 heartbeater.cc:511] Master 127.0.96.254:38383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:28:40.754138 3491 ts_manager.cc:194] Registered new tserver with Master: b83f9030df0f4c20b56b15888dd27338 (127.0.96.194:46711)
I20260620 23:28:40.755210 3491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:36373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:40.784578 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38383
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:41.198050 3820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:41.198412 3820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:41.198525 3820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:41.206784 3820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:41.206951 3820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:41.207083 3820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:41.218390 3820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:41.220093 3820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:41.221732 3820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:28:41.235819 3680 heartbeater.cc:503] Master 127.0.96.254:38383 was elected leader, sending a full tablet report...
W20260620 23:28:41.239686 3825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:41.240136 3826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:41.241001 3828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:41.241344 3820 server_base.cc:1034] running on GCE node
I20260620 23:28:41.242175 3820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:41.243460 3820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:41.244774 3820 hybrid_clock.cc:648] HybridClock initialized: now 1781998121244704 us; error 50 us; skew 500 ppm
I20260620 23:28:41.245303 3820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:41.249684 3820 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:41.251752 3820 webserver.cc:460] Webserver started at http://127.0.96.195:43931/ using document root <none> and password file <none>
I20260620 23:28:41.252750 3820 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:41.252934 3820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:41.253423 3820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:41.257527 3820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "5beb827e46244a94a56e4db4b5c544bc"
format_stamp: "Formatted at 2026-06-20 23:28:41 on dist-test-slave-v34r"
server_key: "8ca11e1edc44380cb66697622f409e39"
server_key_iv: "621731a742a4956411068f7541e86456"
server_key_version: "encryptionkey@0"
I20260620 23:28:41.258605 3820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "5beb827e46244a94a56e4db4b5c544bc"
format_stamp: "Formatted at 2026-06-20 23:28:41 on dist-test-slave-v34r"
server_key: "8ca11e1edc44380cb66697622f409e39"
server_key_iv: "621731a742a4956411068f7541e86456"
server_key_version: "encryptionkey@0"
I20260620 23:28:41.265743 3820 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 23:28:41.270435 3835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:41.272754 3820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:41.273025 3820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "5beb827e46244a94a56e4db4b5c544bc"
format_stamp: "Formatted at 2026-06-20 23:28:41 on dist-test-slave-v34r"
server_key: "8ca11e1edc44380cb66697622f409e39"
server_key_iv: "621731a742a4956411068f7541e86456"
server_key_version: "encryptionkey@0"
I20260620 23:28:41.273298 3820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:41.290419 3820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:41.296017 3820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:41.296423 3820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:41.297825 3820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:41.299574 3820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:41.299702 3820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:41.299871 3820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:41.299952 3820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:41.339419 3820 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:40115
I20260620 23:28:41.339468 3947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:40115 every 8 connection(s)
I20260620 23:28:41.341159 3820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:41.349327 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3820
I20260620 23:28:41.349546 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:41.350063 387 external_mini_cluster.cc:1428] Setting key a68b3434f66e12269c4cbd48056ab413
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:41.366036 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:41.344329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:36985 (local address 127.0.96.254:38383)
0620 23:28:41.344910 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:41.344919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:41.346084 (+ 1165us) server_negotiation.cc:407] Connection header received
0620 23:28:41.348230 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:41.348235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:41.348345 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:41.348473 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:41.350769 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.351846 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.353924 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.354376 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.354543 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:41.358798 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:41.358804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:41.358807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:41.358849 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:41.362225 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.363246 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.363251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.363254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.363387 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.363879 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.363886 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.363894 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.364211 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:41.364389 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:41.365341 (+ 952us) server_negotiation.cc:299] Negotiation successful
0620 23:28:41.365606 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":270,"threads_started":1}
I20260620 23:28:41.367339 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:41.344793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38383 (local address 127.0.96.195:36985)
0620 23:28:41.345838 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:41.345911 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:41.347865 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:41.348765 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:41.348776 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:41.349609 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:41.350502 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:41.350527 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.352049 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.352054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:41.353741 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:41.353757 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.354645 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.354654 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:41.354944 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:41.355771 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:41.355799 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:41.358456 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:41.362474 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:41.362485 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:41.362506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:41.363075 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:41.363510 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:41.363515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:41.363521 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:41.363747 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:41.364349 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:41.364360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:41.364844 (+ 484us) client_negotiation.cc:769] Sending connection context
0620 23:28:41.365212 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 23:28:41.365901 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":432,"threads_started":1}
I20260620 23:28:41.369167 3948 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38383
I20260620 23:28:41.369632 3948 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:41.370524 3948 heartbeater.cc:511] Master 127.0.96.254:38383 requested a full tablet report, sending...
I20260620 23:28:41.372238 3491 ts_manager.cc:194] Registered new tserver with Master: 5beb827e46244a94a56e4db4b5c544bc (127.0.96.195:40115)
I20260620 23:28:41.373226 3491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:36985
I20260620 23:28:41.378688 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:41.429641 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:41.414911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.0.96.254:38383)
0620 23:28:41.415201 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:41.415210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:41.415679 (+ 469us) server_negotiation.cc:407] Connection header received
0620 23:28:41.415931 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:41.415935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:41.416004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:41.416114 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:41.417618 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.418387 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.419616 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.420010 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.420145 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:41.424156 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:41.424161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:41.424165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:41.424204 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:41.426875 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.427602 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.427614 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.427616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.427733 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.428130 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.428137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.428140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.428472 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:41.428723 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:41.429055 (+ 332us) server_negotiation.cc:299] Negotiation successful
0620 23:28:41.429280 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 23:28:41.430512 3491 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41528
I20260620 23:28:41.432435 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3552
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:41.450167 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:41.433596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.0.96.193:39733)
0620 23:28:41.434325 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:41.434333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:41.434368 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:28:41.434442 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:41.434449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:41.434713 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:41.434882 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:41.436065 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.437313 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.438578 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.439274 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.439407 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:41.443668 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:41.443688 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:41.443710 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:41.443769 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:41.447027 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.447938 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.447947 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.447950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.448115 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.448526 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.448530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.448531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.448956 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:41.449312 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:41.449567 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 23:28:41.449820 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
W20260620 23:28:41.451716 3675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51328
W20260620 23:28:41.453179 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3552: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:41.453430 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3552
I20260620 23:28:41.471755 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3686
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:41.491088 3961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:41.472934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.0.96.194:46711)
0620 23:28:41.473635 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:41.473642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:41.473667 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:28:41.473733 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:41.473740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:41.473971 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:41.474120 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:41.475580 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.476766 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.478437 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.479102 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.479260 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:41.484572 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:41.484587 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:41.484603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:41.484644 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:41.487794 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.488702 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.488710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.488713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.488849 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.489386 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.489391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.489395 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.489845 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:41.490095 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:41.490463 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:28:41.490671 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":324,"threads_started":1}
W20260620 23:28:41.492792 3809 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37014
W20260620 23:28:41.494964 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3686: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:41.495169 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3686
I20260620 23:28:41.512771 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3820
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998121, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:41.534848 3962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:41.514128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.0.96.195:40115)
0620 23:28:41.515231 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:41.515248 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:41.515305 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:28:41.515407 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:41.515416 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:41.515718 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:41.515925 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:41.517893 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.519118 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.520839 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:41.521714 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:41.522093 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:41.527362 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:41.527386 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:41.527408 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:41.527468 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:41.531189 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.532203 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.532213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.532216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.532393 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:41.532908 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:41.532912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:41.532914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:41.533455 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:41.533753 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:41.534081 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 23:28:41.534372 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":663,"threads_started":1}
W20260620 23:28:41.536638 3943 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41246
W20260620 23:28:41.538331 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3820: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:28:41.538610 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3820
I20260620 23:28:41.556672 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3459
I20260620 23:28:41.558199 3521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:41.678895 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3459
2026-06-20T23:28:41Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4018 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3966](info): set up 2 sockets
Jun 20 23:28:41 dist-test-slave-v34r krb5kdc[3966](info): commencing operation
krb5kdc: starting...
W20260620 23:28:42.833742 387 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:42 dist-test-slave-v34r krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:43.064708 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:45087
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:45087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:43.416082 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:43.416407 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:43.416594 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:43.424010 3979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:43.424180 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:43.424266 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:43.424386 3979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:43.424484 3979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:43.434473 3979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:45087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:45087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:43.435945 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:43.437376 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:43.449985 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:43.449846 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:43.449928 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:43.451522 3979 server_base.cc:1034] running on GCE node
I20260620 23:28:43.452582 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:43.454538 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:43.455821 3979 hybrid_clock.cc:648] HybridClock initialized: now 1781998123455740 us; error 60 us; skew 500 ppm
I20260620 23:28:43.456252 3979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:43 dist-test-slave-v34r krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:43.466144 3979 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:43.468348 3979 webserver.cc:460] Webserver started at http://127.0.96.254:34635/ using document root <none> and password file <none>
I20260620 23:28:43.469399 3979 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:43.469581 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:43.469995 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:43.474090 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "0ecfd140f12047ac95684e4054988a2b"
format_stamp: "Formatted at 2026-06-20 23:28:43 on dist-test-slave-v34r"
server_key: "53d25e9eb65cc8d00e8a2ba4e41bcc6e"
server_key_iv: "c91dc2fd2e2ce6fcc10f448f8b6195bd"
server_key_version: "encryptionkey@0"
I20260620 23:28:43.475111 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "0ecfd140f12047ac95684e4054988a2b"
format_stamp: "Formatted at 2026-06-20 23:28:43 on dist-test-slave-v34r"
server_key: "53d25e9eb65cc8d00e8a2ba4e41bcc6e"
server_key_iv: "c91dc2fd2e2ce6fcc10f448f8b6195bd"
server_key_version: "encryptionkey@0"
I20260620 23:28:43.482470 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 23:28:43.486969 3994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:43.488863 3979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 23:28:43.489144 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "0ecfd140f12047ac95684e4054988a2b"
format_stamp: "Formatted at 2026-06-20 23:28:43 on dist-test-slave-v34r"
server_key: "53d25e9eb65cc8d00e8a2ba4e41bcc6e"
server_key_iv: "c91dc2fd2e2ce6fcc10f448f8b6195bd"
server_key_version: "encryptionkey@0"
I20260620 23:28:43.489413 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:43.505760 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:43.511059 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:43.511430 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:43.534623 3979 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:45087
I20260620 23:28:43.534657 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:45087 every 8 connection(s)
I20260620 23:28:43.536466 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:43.539685 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 3979
I20260620 23:28:43.539953 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:43.540694 387 external_mini_cluster.cc:1428] Setting key 79f874b49c76e2fa24a0018ece31e644
I20260620 23:28:43.540772 4046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:43.556210 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b: Bootstrap starting.
Jun 20 23:28:43 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:43.560273 4046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:43.561766 4046 log.cc:824] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:43.565469 4046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b: No bootstrap required, opened a new log
I20260620 23:28:43.568189 4049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:43.543119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.0.96.254:45087)
0620 23:28:43.544157 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:43.544172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:43.544231 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:28:43.546214 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:43.546244 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:43.546789 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:43.547251 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:43.549174 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:43.550600 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:43.552552 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:43.553521 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:43.553716 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:43.558872 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:43.558885 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:43.558906 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:43.558986 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:43.563178 (+ 4192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:43.564119 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:43.564128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:43.564136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:43.564341 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:43.564738 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:43.564742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:43.564745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:43.565503 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:43.565798 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:43.566336 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 23:28:43.566793 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":466,"threads_started":1}
I20260620 23:28:43.574739 4046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecfd140f12047ac95684e4054988a2b" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45087 } }
I20260620 23:28:43.575094 4046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:43.575161 4046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ecfd140f12047ac95684e4054988a2b, State: Initialized, Role: FOLLOWER
I20260620 23:28:43.575708 4046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecfd140f12047ac95684e4054988a2b" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45087 } }
I20260620 23:28:43.575959 4046 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:43.576114 4046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:43.576321 4046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:43.578867 4046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecfd140f12047ac95684e4054988a2b" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45087 } }
I20260620 23:28:43.579331 4046 leader_election.cc:302] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ecfd140f12047ac95684e4054988a2b; no voters:
I20260620 23:28:43.579923 4046 leader_election.cc:288] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:43.580171 4051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:43.581902 4051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [term 1 LEADER]: Becoming Leader. State: Replica: 0ecfd140f12047ac95684e4054988a2b, State: Running, Role: LEADER
I20260620 23:28:43.582554 4051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecfd140f12047ac95684e4054988a2b" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45087 } }
I20260620 23:28:43.582958 4046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:43.586714 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ecfd140f12047ac95684e4054988a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecfd140f12047ac95684e4054988a2b" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45087 } } }
I20260620 23:28:43.586687 4053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ecfd140f12047ac95684e4054988a2b. Latest consensus state: current_term: 1 leader_uuid: "0ecfd140f12047ac95684e4054988a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ecfd140f12047ac95684e4054988a2b" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45087 } } }
I20260620 23:28:43.587285 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:43.587306 4053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:43.592948 4060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:43.597766 4060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:43.606191 4060 catalog_manager.cc:1347] Generated new cluster ID: 06ea6e55c73f427688bc864bff3147d2
I20260620 23:28:43.606329 4060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:43.616554 4060 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:43.618827 4060 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:43.630611 4060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ecfd140f12047ac95684e4054988a2b: Generated new TSK 0
I20260620 23:28:43.632079 4060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:43.692330 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45087
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:44.067489 4072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:44.067889 4072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:44.068081 4072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:44.076520 4072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:44.076684 4072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:44.076910 4072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:44.088959 4072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:44.091018 4072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:44.092799 4072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:44.110416 4077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:44.110476 4080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:44.110500 4078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:44.112864 4072 server_base.cc:1034] running on GCE node
I20260620 23:28:44.113778 4072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:44.115180 4072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:44.116468 4072 hybrid_clock.cc:648] HybridClock initialized: now 1781998124116410 us; error 59 us; skew 500 ppm
I20260620 23:28:44.116896 4072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:44 dist-test-slave-v34r krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:44.121129 4072 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:44.123548 4072 webserver.cc:460] Webserver started at http://127.0.96.193:34591/ using document root <none> and password file <none>
I20260620 23:28:44.124563 4072 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:44.124761 4072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:44.125254 4072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:44.129323 4072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "5e742d3042574464908b115b3f45b199"
format_stamp: "Formatted at 2026-06-20 23:28:44 on dist-test-slave-v34r"
server_key: "a5455d4f7bbb46734c87219f2488c95e"
server_key_iv: "1ce86fc423a7e81840bc9918604886c6"
server_key_version: "encryptionkey@0"
I20260620 23:28:44.130609 4072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "5e742d3042574464908b115b3f45b199"
format_stamp: "Formatted at 2026-06-20 23:28:44 on dist-test-slave-v34r"
server_key: "a5455d4f7bbb46734c87219f2488c95e"
server_key_iv: "1ce86fc423a7e81840bc9918604886c6"
server_key_version: "encryptionkey@0"
I20260620 23:28:44.138200 4072 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:28:44.143222 4087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:44.145634 4072 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:28:44.145934 4072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "5e742d3042574464908b115b3f45b199"
format_stamp: "Formatted at 2026-06-20 23:28:44 on dist-test-slave-v34r"
server_key: "a5455d4f7bbb46734c87219f2488c95e"
server_key_iv: "1ce86fc423a7e81840bc9918604886c6"
server_key_version: "encryptionkey@0"
I20260620 23:28:44.146229 4072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:44.175666 4072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:44.181879 4072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:44.182602 4072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:44.184266 4072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:44.186650 4072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:44.186815 4072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:44.187004 4072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:44.187088 4072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:44.230232 4072 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:45313
I20260620 23:28:44.230298 4199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:45313 every 8 connection(s)
I20260620 23:28:44.231956 4072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:44.236152 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4072
I20260620 23:28:44.236439 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:44.237268 387 external_mini_cluster.cc:1428] Setting key 8f6f776551916c5966ad0bb50ea2e374
Jun 20 23:28:44 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:44.255963 4202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:44.235257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:53835 (local address 127.0.96.254:45087)
0620 23:28:44.235966 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:44.235976 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:44.237051 (+ 1075us) server_negotiation.cc:407] Connection header received
0620 23:28:44.239231 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:44.239239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:44.239348 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:44.239535 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:44.241686 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.242478 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.244091 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.244492 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.244635 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:44.249399 (+ 4764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:44.249405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:44.249409 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:44.249447 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:44.252609 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:44.253578 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:44.253582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:44.253585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:44.253703 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:44.254155 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:44.254162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:44.254170 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:44.254445 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:44.254600 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:44.255355 (+ 755us) server_negotiation.cc:299] Negotiation successful
0620 23:28:44.255592 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":321,"threads_started":1}
I20260620 23:28:44.257066 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:44.235858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45087 (local address 127.0.96.193:53835)
0620 23:28:44.236761 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:44.236813 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:44.238895 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:44.239973 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:44.239983 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:44.240610 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:44.241464 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:44.241483 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.242646 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.242658 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:44.243936 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:44.243949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.244732 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.244741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:44.245064 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:44.245819 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:44.245843 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:44.249120 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:44.252798 (+ 3678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:44.252809 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:44.252830 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:44.253432 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:44.253818 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:44.253822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:44.253826 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:44.254033 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:44.254574 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:44.254583 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:44.254952 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 23:28:44.255234 (+ 282us) client_negotiation.cc:240] Negotiation successful
0620 23:28:44.255816 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":370,"threads_started":1}
I20260620 23:28:44.259007 4200 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45087
I20260620 23:28:44.259557 4200 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:28:44.260767 4200 heartbeater.cc:511] Master 127.0.96.254:45087 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:28:44.263842 4011 ts_manager.cc:194] Registered new tserver with Master: 5e742d3042574464908b115b3f45b199 (127.0.96.193:45313)
I20260620 23:28:44.265743 4011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:53835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:44.290452 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45087
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:44.698165 4206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:44.698567 4206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:44.698756 4206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:44.707473 4206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:44.707638 4206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:44.707858 4206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:44.719710 4206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:44.721833 4206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:44.723820 4206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:44.741225 4211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:44.741267 4212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:44.741294 4214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:44.742156 4206 server_base.cc:1034] running on GCE node
I20260620 23:28:44.742929 4206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:44.744057 4206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:44.745275 4206 hybrid_clock.cc:648] HybridClock initialized: now 1781998124745199 us; error 78 us; skew 500 ppm
I20260620 23:28:44.745647 4206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:44 dist-test-slave-v34r krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:44.749320 4206 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:44.751195 4206 webserver.cc:460] Webserver started at http://127.0.96.194:34213/ using document root <none> and password file <none>
I20260620 23:28:44.752056 4206 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:44.752216 4206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:44.752593 4206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:44.756024 4206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "aa1fc3f4ef564167a2a0c2b49da3fe36"
format_stamp: "Formatted at 2026-06-20 23:28:44 on dist-test-slave-v34r"
server_key: "25ec3c9078ec6e106e41b160cd8e480f"
server_key_iv: "e82c2de73ffd36d9e6e42d87469c3b21"
server_key_version: "encryptionkey@0"
I20260620 23:28:44.756966 4206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "aa1fc3f4ef564167a2a0c2b49da3fe36"
format_stamp: "Formatted at 2026-06-20 23:28:44 on dist-test-slave-v34r"
server_key: "25ec3c9078ec6e106e41b160cd8e480f"
server_key_iv: "e82c2de73ffd36d9e6e42d87469c3b21"
server_key_version: "encryptionkey@0"
I20260620 23:28:44.763128 4206 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.003s
I20260620 23:28:44.767241 4221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:44.769152 4206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:28:44.769320 4206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "aa1fc3f4ef564167a2a0c2b49da3fe36"
format_stamp: "Formatted at 2026-06-20 23:28:44 on dist-test-slave-v34r"
server_key: "25ec3c9078ec6e106e41b160cd8e480f"
server_key_iv: "e82c2de73ffd36d9e6e42d87469c3b21"
server_key_version: "encryptionkey@0"
I20260620 23:28:44.769563 4206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:44.799232 4206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:44.803875 4206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:44.804200 4206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:44.805352 4206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:44.806836 4206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:44.806914 4206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:44.807101 4206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:44.807157 4206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:44.845546 4206 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:35551
I20260620 23:28:44.845669 4333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:35551 every 8 connection(s)
I20260620 23:28:44.846951 4206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:44.855278 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4206
I20260620 23:28:44.855626 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:44.856137 387 external_mini_cluster.cc:1428] Setting key 0fc616ba52c6443a446b9b4ae7a46225
Jun 20 23:28:44 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:44.873688 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:44.850251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:40169 (local address 127.0.96.254:45087)
0620 23:28:44.850792 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:44.850798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:44.851893 (+ 1095us) server_negotiation.cc:407] Connection header received
0620 23:28:44.854174 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:44.854180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:44.854306 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:44.854440 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:44.856475 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.857480 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.860037 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.860522 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.860653 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:44.865520 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:44.865549 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:44.865561 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:44.865618 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:44.869568 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:44.870645 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:44.870653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:44.870660 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:44.870857 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:44.871324 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:44.871333 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:44.871339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:44.871806 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:44.871972 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:44.872812 (+ 840us) server_negotiation.cc:299] Negotiation successful
0620 23:28:44.873160 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":291,"threads_started":1}
I20260620 23:28:44.875049 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:44.850757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45087 (local address 127.0.96.194:40169)
0620 23:28:44.851684 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:44.851735 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:44.853837 (+ 2102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:44.854735 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:44.854745 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:44.855429 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:44.856278 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:44.856306 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.857708 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.857714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:44.859811 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:44.859838 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:44.860789 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:44.860800 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:44.861216 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:44.862076 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:44.862113 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:44.865204 (+ 3091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:44.869795 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:44.869821 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:44.869839 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:44.870491 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:44.870985 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:44.870988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:44.870991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:44.871195 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:44.871962 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:44.871970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:44.872372 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 23:28:44.872660 (+ 288us) client_negotiation.cc:240] Negotiation successful
0620 23:28:44.873383 (+ 723us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":406,"threads_started":1}
I20260620 23:28:44.877009 4334 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45087
I20260620 23:28:44.877529 4334 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:44.878748 4334 heartbeater.cc:511] Master 127.0.96.254:45087 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:28:44.881213 4011 ts_manager.cc:194] Registered new tserver with Master: aa1fc3f4ef564167a2a0c2b49da3fe36 (127.0.96.194:35551)
I20260620 23:28:44.882309 4011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:40169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:44.912011 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45087
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:40095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:28:45.270347 4200 heartbeater.cc:503] Master 127.0.96.254:45087 was elected leader, sending a full tablet report...
W20260620 23:28:45.324564 4340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:45.325057 4340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:45.325240 4340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:45.333580 4340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:45.333750 4340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:45.333988 4340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:45.345798 4340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:40095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45087
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:45.347620 4340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:45.349277 4340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:45.367079 4348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:45.367139 4346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:45.367139 4345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:45.368451 4340 server_base.cc:1034] running on GCE node
I20260620 23:28:45.369385 4340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:45.370623 4340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:45.371903 4340 hybrid_clock.cc:648] HybridClock initialized: now 1781998125371798 us; error 78 us; skew 500 ppm
I20260620 23:28:45.372367 4340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:45.376899 4340 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:45.379112 4340 webserver.cc:460] Webserver started at http://127.0.96.195:38777/ using document root <none> and password file <none>
I20260620 23:28:45.380051 4340 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:45.380223 4340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:45.380606 4340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:45.384619 4340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "5e538f0189a54687af0710d93d314d85"
format_stamp: "Formatted at 2026-06-20 23:28:45 on dist-test-slave-v34r"
server_key: "3f05dbd6f47fc5051921bf76adbb6c89"
server_key_iv: "6fa8a51d8881114984ecd412bf5656b4"
server_key_version: "encryptionkey@0"
I20260620 23:28:45.385699 4340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "5e538f0189a54687af0710d93d314d85"
format_stamp: "Formatted at 2026-06-20 23:28:45 on dist-test-slave-v34r"
server_key: "3f05dbd6f47fc5051921bf76adbb6c89"
server_key_iv: "6fa8a51d8881114984ecd412bf5656b4"
server_key_version: "encryptionkey@0"
I20260620 23:28:45.392498 4340 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260620 23:28:45.397537 4355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:45.399626 4340 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:45.399936 4340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "5e538f0189a54687af0710d93d314d85"
format_stamp: "Formatted at 2026-06-20 23:28:45 on dist-test-slave-v34r"
server_key: "3f05dbd6f47fc5051921bf76adbb6c89"
server_key_iv: "6fa8a51d8881114984ecd412bf5656b4"
server_key_version: "encryptionkey@0"
I20260620 23:28:45.400249 4340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:45.417605 4340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:45.423318 4340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:45.423636 4340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:45.424829 4340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:45.426625 4340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:45.426712 4340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:45.426822 4340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:45.426882 4340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:45.467346 4340 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:40209
I20260620 23:28:45.467396 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:40209 every 8 connection(s)
I20260620 23:28:45.469110 4340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:45.479492 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4340
I20260620 23:28:45.479723 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:45.480144 387 external_mini_cluster.cc:1428] Setting key 152ff1fcde55ef2f330b955c879146a3
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:45.494087 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.472073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38427 (local address 127.0.96.254:45087)
0620 23:28:45.472638 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:45.472643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:45.474042 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 23:28:45.476327 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:45.476334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:45.476466 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:45.476603 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:45.478608 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.479459 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.481713 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.482336 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.482499 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.486984 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:45.486990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:45.486994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:45.487034 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:45.490161 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.491206 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.491219 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.491222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.491362 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.491859 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.491864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.491868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.492207 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:45.492391 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:45.493334 (+ 943us) server_negotiation.cc:299] Negotiation successful
0620 23:28:45.493629 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260620 23:28:45.495675 4471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.472696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45087 (local address 127.0.96.195:38427)
0620 23:28:45.473813 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:45.473876 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:45.475958 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:45.476917 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:45.476929 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:45.477661 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:45.478384 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:45.478405 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.479638 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.479664 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:45.481536 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:45.481554 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.482635 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.482653 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:45.483029 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.483943 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:45.483971 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:45.486679 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:45.490374 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:45.490389 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:45.490410 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:45.491037 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:45.491488 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:45.491493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:45.491497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:45.491726 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:45.492343 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:45.492354 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:45.492828 (+ 474us) client_negotiation.cc:769] Sending connection context
0620 23:28:45.493205 (+ 377us) client_negotiation.cc:240] Negotiation successful
0620 23:28:45.493952 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":513,"threads_started":1}
I20260620 23:28:45.497839 4468 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45087
I20260620 23:28:45.498281 4468 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:45.499215 4468 heartbeater.cc:511] Master 127.0.96.254:45087 requested a full tablet report, sending...
I20260620 23:28:45.501330 4010 ts_manager.cc:194] Registered new tserver with Master: 5e538f0189a54687af0710d93d314d85 (127.0.96.195:40209)
I20260620 23:28:45.502285 4010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:38427
I20260620 23:28:45.509402 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:45.562296 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.544252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.0.96.193:45313)
0620 23:28:45.545125 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:45.545135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:45.545172 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:28:45.545251 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:45.545260 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:45.545539 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:45.545738 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:45.547119 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.548257 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.549773 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.550495 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.550639 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.555063 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:45.555082 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:45.555103 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:45.555164 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:45.558871 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.559822 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.559833 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.559837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.560043 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.560485 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.560488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.560490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.560969 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:45.561333 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:45.561663 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 23:28:45.561951 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":396,"threads_started":1}
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:45.584829 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.568241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.0.96.254:45087)
0620 23:28:45.568516 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:45.568523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:45.568722 (+ 199us) server_negotiation.cc:407] Connection header received
0620 23:28:45.568963 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:45.568970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:45.569104 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:45.569229 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:45.570714 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.571499 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.572863 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.573442 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.573594 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.578341 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:45.578347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:45.578351 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:45.578400 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:45.581355 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.582285 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.582291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.582294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.582511 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.582971 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.582976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.583000 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.583487 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:45.583719 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:45.584295 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 23:28:45.584526 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 23:28:45.585594 4010 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37150
I20260620 23:28:45.586910 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4072
I20260620 23:28:45.602191 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.588088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.0.96.193:45313)
0620 23:28:45.588386 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:45.588394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:45.588521 (+ 127us) server_negotiation.cc:407] Connection header received
0620 23:28:45.588703 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:45.588708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:45.588831 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:45.588976 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:45.590366 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.591166 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.592398 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.592898 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.593213 (+ 315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.596089 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:45.596097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:45.596101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:45.596149 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:45.599388 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.600231 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.600236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.600239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.600436 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.600815 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.600821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.600824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.601189 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:45.601431 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:45.601736 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 23:28:45.601936 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 23:28:45.603024 4195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:45.690047 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4072
I20260620 23:28:45.710830 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4206
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:45.730160 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.711988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.0.96.194:35551)
0620 23:28:45.712730 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:45.712739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:45.712773 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:28:45.712853 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:45.712861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:45.713299 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:45.713473 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:45.714746 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.715924 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.717551 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.718259 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.718389 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.723063 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:45.723082 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:45.723104 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:45.723155 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:45.726629 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.727450 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.727459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.727462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.727631 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.728123 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.728128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.728130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.728623 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:45.728961 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:45.729348 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 23:28:45.729714 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":410,"threads_started":1}
I20260620 23:28:45.731922 4329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:45.825536 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4206
I20260620 23:28:45.842581 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4340
Jun 20 23:28:45 dist-test-slave-v34r krb5kdc[3966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:45.861119 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:45.843864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.96.195:40209)
0620 23:28:45.844675 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:45.844682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:45.844713 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:28:45.844778 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:45.844785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:45.845111 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:45.845328 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:45.846658 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.847684 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.848975 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:45.849603 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:45.849748 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:45.854563 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:45.854585 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:45.854607 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:45.854659 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:45.858002 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.858842 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.858852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.858854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.858998 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:45.859361 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:45.859364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:45.859366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:45.859771 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:45.860030 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:45.860328 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:28:45.860572 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":435,"threads_started":1}
I20260620 23:28:45.862697 4463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:45.950371 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4340
I20260620 23:28:45.968950 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 3979
I20260620 23:28:45.970424 4041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:46.079172 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 3979
2026-06-20T23:28:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4398 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:28:46 dist-test-slave-v34r krb5kdc[4488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:28:46 dist-test-slave-v34r krb5kdc[4488](info): set up 2 sockets
Jun 20 23:28:46 dist-test-slave-v34r krb5kdc[4488](info): commencing operation
krb5kdc: starting...
W20260620 23:28:47.185648 387 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:28:47 dist-test-slave-v34r krb5kdc[4488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:28:47.420135 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42487
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:47.832432 4501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:47.832864 4501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:47.833140 4501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:47.841584 4501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:47.841744 4501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:47.841838 4501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:47.841956 4501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:47.842042 4501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:47.854151 4501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:47.856061 4501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:47.858032 4501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:47.872490 4506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:47.872534 4509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:47.872586 4507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:47.874153 4501 server_base.cc:1034] running on GCE node
I20260620 23:28:47.875193 4501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:47.877140 4501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:47.878420 4501 hybrid_clock.cc:648] HybridClock initialized: now 1781998127878358 us; error 53 us; skew 500 ppm
I20260620 23:28:47.878911 4501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:47 dist-test-slave-v34r krb5kdc[4488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:47.883245 4501 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:28:47.884932 4501 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:47.885143 4501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:47.885516 4501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:47.889523 4501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "0906d1458f0c4454b276784c427f1bc5"
format_stamp: "Formatted at 2026-06-20 23:28:47 on dist-test-slave-v34r"
server_key: "72efc50441259901d03392a646099988"
server_key_iv: "476589fb48f1f9f165b90af1288d1fd2"
server_key_version: "encryptionkey@0"
I20260620 23:28:47.890513 4501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "0906d1458f0c4454b276784c427f1bc5"
format_stamp: "Formatted at 2026-06-20 23:28:47 on dist-test-slave-v34r"
server_key: "72efc50441259901d03392a646099988"
server_key_iv: "476589fb48f1f9f165b90af1288d1fd2"
server_key_version: "encryptionkey@0"
I20260620 23:28:47.897420 4501 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:28:47.902268 4515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:47.904210 4501 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:47.904507 4501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "0906d1458f0c4454b276784c427f1bc5"
format_stamp: "Formatted at 2026-06-20 23:28:47 on dist-test-slave-v34r"
server_key: "72efc50441259901d03392a646099988"
server_key_iv: "476589fb48f1f9f165b90af1288d1fd2"
server_key_version: "encryptionkey@0"
I20260620 23:28:47.904837 4501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:47.924962 4501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:47.931150 4501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:47.931591 4501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:47.955828 4501 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:42487
I20260620 23:28:47.955803 4566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:42487 every 8 connection(s)
I20260620 23:28:47.957175 4501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:47.961614 4567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:47.966841 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4501
I20260620 23:28:47.967089 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:47.967761 387 external_mini_cluster.cc:1428] Setting key 58c5ef2e6b0fb32bfa19b88c6c23b3a2
I20260620 23:28:47.975488 4567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5: Bootstrap starting.
I20260620 23:28:47.980414 4567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:47.982115 4567 log.cc:824] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:28:47 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:47.986172 4567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5: No bootstrap required, opened a new log
I20260620 23:28:47.995364 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:47.970213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.0.96.254:42487)
0620 23:28:47.971211 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:47.971237 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:47.971289 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:28:47.973467 (+ 2178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:47.973495 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:47.973983 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:47.974381 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:47.976748 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:47.978424 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:47.980490 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:47.981467 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:47.981631 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:47.986943 (+ 5312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:47.986952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:47.986973 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:47.987019 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:47.990740 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:47.991528 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:47.991535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:47.991542 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:47.991727 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:47.992124 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:47.992128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:47.992130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:47.992801 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:47.993161 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:47.993730 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 23:28:47.994179 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":466,"threads_started":1}
I20260620 23:28:47.996075 4567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0906d1458f0c4454b276784c427f1bc5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42487 } }
I20260620 23:28:47.996651 4567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:47.996774 4567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0906d1458f0c4454b276784c427f1bc5, State: Initialized, Role: FOLLOWER
I20260620 23:28:47.997589 4567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0906d1458f0c4454b276784c427f1bc5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42487 } }
I20260620 23:28:47.997889 4567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:47.998042 4567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:47.998261 4567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:48.001315 4567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0906d1458f0c4454b276784c427f1bc5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42487 } }
I20260620 23:28:48.002105 4567 leader_election.cc:302] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0906d1458f0c4454b276784c427f1bc5; no voters:
I20260620 23:28:48.002828 4567 leader_election.cc:288] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:48.003053 4572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:48.004069 4572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 0906d1458f0c4454b276784c427f1bc5, State: Running, Role: LEADER
I20260620 23:28:48.004721 4572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0906d1458f0c4454b276784c427f1bc5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42487 } }
I20260620 23:28:48.005808 4567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:48.008497 4573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0906d1458f0c4454b276784c427f1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0906d1458f0c4454b276784c427f1bc5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42487 } } }
I20260620 23:28:48.008396 4574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0906d1458f0c4454b276784c427f1bc5. Latest consensus state: current_term: 1 leader_uuid: "0906d1458f0c4454b276784c427f1bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0906d1458f0c4454b276784c427f1bc5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42487 } } }
I20260620 23:28:48.009192 4574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:48.009047 4573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:48.010622 4579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:48.017505 4579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:48.035794 4579 catalog_manager.cc:1347] Generated new cluster ID: 5099f38c4904431cb91dc9a80444d30d
I20260620 23:28:48.035991 4579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:48.049378 4579 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:48.051450 4579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:48.064149 4579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5: Generated new TSK 0
I20260620 23:28:48.065459 4579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:28:48.129123 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42487
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:48.551167 4593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:48.551559 4593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:48.551676 4593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:48.560020 4593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:48.560180 4593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:48.560321 4593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:48.571700 4593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:48.573572 4593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:48.575220 4593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:48.593274 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:48.593384 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:48.593361 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:48.595973 4593 server_base.cc:1034] running on GCE node
I20260620 23:28:48.596761 4593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:48.597976 4593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:48.599244 4593 hybrid_clock.cc:648] HybridClock initialized: now 1781998128599185 us; error 67 us; skew 500 ppm
I20260620 23:28:48.599654 4593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:48 dist-test-slave-v34r krb5kdc[4488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:48.607218 4593 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:28:48.608772 4593 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:48.608875 4593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:48.609308 4593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:48.612545 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "c542daa830124fca86afc976060444a7"
format_stamp: "Formatted at 2026-06-20 23:28:48 on dist-test-slave-v34r"
server_key: "a9240af3a820f64988b7b97f13d9172f"
server_key_iv: "bdada6972f9b68ee8a9c194e425d3046"
server_key_version: "encryptionkey@0"
I20260620 23:28:48.613493 4593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "c542daa830124fca86afc976060444a7"
format_stamp: "Formatted at 2026-06-20 23:28:48 on dist-test-slave-v34r"
server_key: "a9240af3a820f64988b7b97f13d9172f"
server_key_iv: "bdada6972f9b68ee8a9c194e425d3046"
server_key_version: "encryptionkey@0"
I20260620 23:28:48.619362 4593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:28:48.623433 4607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:48.625353 4593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:28:48.625551 4593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "c542daa830124fca86afc976060444a7"
format_stamp: "Formatted at 2026-06-20 23:28:48 on dist-test-slave-v34r"
server_key: "a9240af3a820f64988b7b97f13d9172f"
server_key_iv: "bdada6972f9b68ee8a9c194e425d3046"
server_key_version: "encryptionkey@0"
I20260620 23:28:48.625792 4593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:48.649307 4593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:48.653707 4593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:48.653996 4593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:48.655192 4593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:48.656693 4593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:48.656786 4593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:48.656906 4593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:48.657049 4593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:48.694414 4593 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:40721
I20260620 23:28:48.694512 4719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:40721 every 8 connection(s)
I20260620 23:28:48.695331 4593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:48.696425 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4593
I20260620 23:28:48.696693 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:48.697600 387 external_mini_cluster.cc:1428] Setting key 830e20d9820adc63a29d935539f33d05
Jun 20 23:28:48 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:28:48.723098 4722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:48.698805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:47557 (local address 127.0.96.254:42487)
0620 23:28:48.699393 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:48.699399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:48.700682 (+ 1283us) server_negotiation.cc:407] Connection header received
0620 23:28:48.702949 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:48.702954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:48.703066 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:48.703196 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:48.705121 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:48.706105 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:48.709170 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:48.709774 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:48.709941 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:48.715603 (+ 5662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:48.715608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:48.715612 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:48.715655 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:48.718698 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:48.720163 (+ 1465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:48.720168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:48.720170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:48.720306 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:48.720955 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:48.720959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:48.720961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:48.721340 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:48.721480 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:48.722608 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0620 23:28:48.722829 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":298,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:28:48.724726 4723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:48.699412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42487 (local address 127.0.96.193:47557)
0620 23:28:48.700469 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:48.700526 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:48.702582 (+ 2056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:48.703495 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:48.703506 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:48.704148 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:48.704872 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:48.704893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:48.706303 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:48.706308 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:48.708814 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:48.708845 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:48.710120 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:48.710131 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:48.710543 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:48.711506 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:48.711546 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:48.715182 (+ 3636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:48.718929 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:48.718947 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:48.718976 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:48.719945 (+ 969us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:48.720453 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:48.720462 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:48.720470 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:48.720789 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:48.721506 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:48.721517 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:48.722142 (+ 625us) client_negotiation.cc:769] Sending connection context
0620 23:28:48.722449 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 23:28:48.723066 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":421,"threads_started":1}
I20260620 23:28:48.726760 4720 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42487
I20260620 23:28:48.727172 4720 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:48.728020 4720 heartbeater.cc:511] Master 127.0.96.254:42487 requested a full tablet report, sending...
I20260620 23:28:48.730824 4532 ts_manager.cc:194] Registered new tserver with Master: c542daa830124fca86afc976060444a7 (127.0.96.193:40721)
I20260620 23:28:48.733549 4532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:47557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:28:48.752270 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42487
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:49.163478 4726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:49.163784 4726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:49.163882 4726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:49.171021 4726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:49.171139 4726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:49.171252 4726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:49.180807 4726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:49.182260 4726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:49.183645 4726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:49.197963 4731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:49.198204 4732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:49.198086 4734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:49.199982 4726 server_base.cc:1034] running on GCE node
I20260620 23:28:49.200809 4726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:49.201954 4726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:49.203222 4726 hybrid_clock.cc:648] HybridClock initialized: now 1781998129203165 us; error 49 us; skew 500 ppm
I20260620 23:28:49.203675 4726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:49 dist-test-slave-v34r krb5kdc[4488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:49.207572 4726 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:28:49.208952 4726 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:49.209223 4726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:49.209625 4726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:49.212728 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "0e1134f95377494c96c73691b7649d5d"
format_stamp: "Formatted at 2026-06-20 23:28:49 on dist-test-slave-v34r"
server_key: "a4fd17d6a4ea4eef5f8ca507618f2ea4"
server_key_iv: "2800d74290461c17ecff3cc1236d4a7a"
server_key_version: "encryptionkey@0"
I20260620 23:28:49.213682 4726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "0e1134f95377494c96c73691b7649d5d"
format_stamp: "Formatted at 2026-06-20 23:28:49 on dist-test-slave-v34r"
server_key: "a4fd17d6a4ea4eef5f8ca507618f2ea4"
server_key_iv: "2800d74290461c17ecff3cc1236d4a7a"
server_key_version: "encryptionkey@0"
I20260620 23:28:49.219677 4726 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260620 23:28:49.223963 4740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:49.225871 4726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 23:28:49.226089 4726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "0e1134f95377494c96c73691b7649d5d"
format_stamp: "Formatted at 2026-06-20 23:28:49 on dist-test-slave-v34r"
server_key: "a4fd17d6a4ea4eef5f8ca507618f2ea4"
server_key_iv: "2800d74290461c17ecff3cc1236d4a7a"
server_key_version: "encryptionkey@0"
I20260620 23:28:49.226329 4726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:49.252687 4726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:49.257694 4726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:49.257994 4726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:49.259119 4726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:49.260579 4726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:49.260708 4726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:49.260848 4726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:49.260941 4726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:49.299871 4726 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:44281
I20260620 23:28:49.299973 4852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:44281 every 8 connection(s)
I20260620 23:28:49.300812 4726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:49.308542 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4726
I20260620 23:28:49.308760 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:49.309289 387 external_mini_cluster.cc:1428] Setting key 8ed73dfc8ec064c575a68f2d4ba5048e
Jun 20 23:28:49 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:49.325258 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.304051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:57627 (local address 127.0.96.254:42487)
0620 23:28:49.304591 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.304598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.305790 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 23:28:49.308083 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.308089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.308238 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.308380 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:49.310651 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.311482 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.313421 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.313888 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.314039 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.318383 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:49.318390 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:49.318394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:49.318437 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:49.321484 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:49.322480 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:49.322484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:49.322487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:49.322625 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:49.323130 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:49.323134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:49.323144 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:49.323470 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:49.323639 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.324509 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.324776 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":253,"threads_started":1}
I20260620 23:28:49.326606 4856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.304560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42487 (local address 127.0.96.194:57627)
0620 23:28:49.305532 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.305625 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.307713 (+ 2088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.308703 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.308721 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.309671 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:49.310420 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.310452 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.311672 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.311677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.313204 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.313220 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.314158 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.314165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.314432 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.315266 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:49.315293 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:49.318061 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:49.321695 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:49.321709 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:49.321742 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:49.322302 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:49.322751 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:49.322757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:49.322761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:49.322997 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:49.323605 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:49.323615 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:49.324077 (+ 462us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.324407 (+ 330us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.325139 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":409,"threads_started":1}
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:28:49.328527 4853 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42487
I20260620 23:28:49.328972 4853 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:28:49.330034 4853 heartbeater.cc:511] Master 127.0.96.254:42487 requested a full tablet report, sending...
I20260620 23:28:49.332281 4532 ts_manager.cc:194] Registered new tserver with Master: 0e1134f95377494c96c73691b7649d5d (127.0.96.194:44281)
I20260620 23:28:49.333630 4532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:57627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:28:49.352559 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42487
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:28:49.687417 4859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:49.687712 4859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:49.687809 4859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:49.695312 4859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:49.695442 4859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:49.695644 4859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:49.705734 4859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:42487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:49.707154 4859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:49.708431 4859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:49.722765 4867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:49.722782 4864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:49.723155 4865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:49.729055 4859 server_base.cc:1034] running on GCE node
I20260620 23:28:49.729799 4859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:49.731052 4859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:49.732321 4859 hybrid_clock.cc:648] HybridClock initialized: now 1781998129732275 us; error 76 us; skew 500 ppm
I20260620 23:28:49.732666 4859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:28:49 dist-test-slave-v34r krb5kdc[4488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:28:49.736171 4859 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:28:49.737645 4859 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:49.737779 4859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:49.738109 4859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:49.738554 4720 heartbeater.cc:503] Master 127.0.96.254:42487 was elected leader, sending a full tablet report...
I20260620 23:28:49.741648 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "a79287431d884dda9f121833fadf37c2"
format_stamp: "Formatted at 2026-06-20 23:28:49 on dist-test-slave-v34r"
server_key: "981ac1790c951cc9aa084509eb98485e"
server_key_iv: "fac529267f4dec631e1e69c62985ebd0"
server_key_version: "encryptionkey@0"
I20260620 23:28:49.742471 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "a79287431d884dda9f121833fadf37c2"
format_stamp: "Formatted at 2026-06-20 23:28:49 on dist-test-slave-v34r"
server_key: "981ac1790c951cc9aa084509eb98485e"
server_key_iv: "fac529267f4dec631e1e69c62985ebd0"
server_key_version: "encryptionkey@0"
I20260620 23:28:49.747685 4859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:28:49.751468 4873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:49.753273 4859 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:28:49.753440 4859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "a79287431d884dda9f121833fadf37c2"
format_stamp: "Formatted at 2026-06-20 23:28:49 on dist-test-slave-v34r"
server_key: "981ac1790c951cc9aa084509eb98485e"
server_key_iv: "fac529267f4dec631e1e69c62985ebd0"
server_key_version: "encryptionkey@0"
I20260620 23:28:49.753639 4859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:49.766675 4859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:49.770885 4859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:49.771129 4859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:49.772149 4859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:49.773732 4859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:49.773830 4859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:49.773902 4859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:49.773947 4859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:49.812299 4859 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:37667
I20260620 23:28:49.812455 4985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:37667 every 8 connection(s)
I20260620 23:28:49.813244 4859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:49.813860 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 4859
I20260620 23:28:49.814136 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:49.814704 387 external_mini_cluster.cc:1428] Setting key b230eb5326bf36e380226f23c1b26274
Jun 20 23:28:49 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:28:49.834972 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.816151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:56189 (local address 127.0.96.254:42487)
0620 23:28:49.816390 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.816395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.818113 (+ 1718us) server_negotiation.cc:407] Connection header received
0620 23:28:49.820086 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.820091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.820179 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.820299 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:49.821904 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.822699 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.824263 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.824653 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.824791 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.828476 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:49.828484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:49.828492 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:49.828539 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:49.831946 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:49.832678 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:49.832684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:49.832687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:49.832809 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:49.833292 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:49.833297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:49.833300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:49.833576 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:49.833746 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.834415 (+ 669us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.834631 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:28:49.835947 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.816833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42487 (local address 127.0.96.195:56189)
0620 23:28:49.817941 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.818004 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.819817 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.820539 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.820548 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.821133 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:49.821747 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.821765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.822866 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.822870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.824100 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.824111 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.824876 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.824883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.825172 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.825894 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:28:49.825916 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:49.828254 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:49.832082 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:49.832092 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:49.832108 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:49.832552 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:49.832914 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:28:49.832921 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:28:49.832924 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:28:49.833176 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:28:49.833708 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:49.833716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:49.834054 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.834316 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.834826 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":446,"threads_started":1}
I20260620 23:28:49.837826 4986 heartbeater.cc:347] Connected to a master server at 127.0.96.254:42487
I20260620 23:28:49.838119 4986 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:49.838704 4986 heartbeater.cc:511] Master 127.0.96.254:42487 requested a full tablet report, sending...
I20260620 23:28:49.840438 4532 ts_manager.cc:194] Registered new tserver with Master: a79287431d884dda9f121833fadf37c2 (127.0.96.195:37667)
I20260620 23:28:49.841467 4532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:56189
I20260620 23:28:49.844298 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:28:49.863255 4855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.849297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.0.96.254:42487)
0620 23:28:49.849489 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.849494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.850139 (+ 645us) server_negotiation.cc:407] Connection header received
0620 23:28:49.850304 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.850307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.850376 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.850499 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:49.852137 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.852856 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.854294 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.854733 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.854844 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.857737 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:49.857742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:49.857746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:49.857780 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:49.860470 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:49.861410 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:49.861414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:49.861417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:49.861585 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:49.861969 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:49.861973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:49.861974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:49.862274 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:49.862462 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.862830 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.862997 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:28:49.868587 4532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:49.877189 4532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:49.907853 4998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.892973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:44281 (local address 127.0.0.1:32858)
0620 23:28:49.894097 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.894145 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.894322 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.896399 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.896411 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.896466 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:49.897010 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.897024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.899233 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.899242 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.904187 (+ 4945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.904211 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.906649 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.906656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.906887 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.906938 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.907098 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.907362 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":33,"thread_start_us":2396,"threads_started":1}
I20260620 23:28:49.907905 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.893163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.0.96.194:44281)
0620 23:28:49.895501 (+ 2338us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.895513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.895557 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:28:49.895707 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.895717 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.896027 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.896390 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:49.897323 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.899061 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.904448 (+ 5387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.906067 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.906214 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.906487 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.907256 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.907505 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2114,"thread_start_us":1996,"threads_started":1}
I20260620 23:28:49.908339 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.893768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40721 (local address 127.0.0.1:44774)
0620 23:28:49.898487 (+ 4719us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.898510 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.898645 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.899417 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.899421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.900871 (+ 1450us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:49.901720 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.901733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.903361 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.903367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.905403 (+ 2036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.905422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.907506 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.907515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.907704 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.907733 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.907822 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.907976 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4569,"mutex_wait_us":3,"thread_start_us":3948,"threads_started":1}
I20260620 23:28:49.909312 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.894388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.0.96.193:40721)
0620 23:28:49.897496 (+ 3108us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.897510 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.898775 (+ 1265us) server_negotiation.cc:407] Connection header received
0620 23:28:49.898888 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.898898 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.899200 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.899428 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:49.901918 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.903216 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.905626 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.907028 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.907162 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.907360 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.908310 (+ 950us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.908569 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2808,"thread_start_us":2530,"threads_started":1}
I20260620 23:28:49.910605 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.893011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37667 (local address 127.0.0.1:53552)
0620 23:28:49.898312 (+ 5301us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.898342 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.898531 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.900008 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.900015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.900050 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:49.901791 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.902339 (+ 548us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.904469 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.904474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.906458 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.906471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.909540 (+ 3069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.909547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.909784 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.909821 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.909993 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.910163 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4952,"mutex_wait_us":251,"thread_start_us":4942,"threads_started":1}
I20260620 23:28:49.910734 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.893970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.0.96.195:37667)
0620 23:28:49.895106 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.895119 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.898695 (+ 3576us) server_negotiation.cc:407] Connection header received
0620 23:28:49.898830 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.898840 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.899206 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.899459 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:49.902614 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.904294 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.906681 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.908656 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.908846 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.909362 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.910130 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.910382 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":684,"threads_started":1}
I20260620 23:28:49.913198 4788 tablet_service.cc:1467] Processing CreateTablet for tablet 25fa31c33b99441089ce76673100ed7c (DEFAULT_TABLE table=test-table [id=03185523c29548139dcaf7d7e5046c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:49.914522 4788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fa31c33b99441089ce76673100ed7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:49.914690 4655 tablet_service.cc:1467] Processing CreateTablet for tablet 25fa31c33b99441089ce76673100ed7c (DEFAULT_TABLE table=test-table [id=03185523c29548139dcaf7d7e5046c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:49.915320 4921 tablet_service.cc:1467] Processing CreateTablet for tablet 25fa31c33b99441089ce76673100ed7c (DEFAULT_TABLE table=test-table [id=03185523c29548139dcaf7d7e5046c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:49.916718 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fa31c33b99441089ce76673100ed7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:49.917200 4921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fa31c33b99441089ce76673100ed7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:49.927023 5004 tablet_bootstrap.cc:492] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Bootstrap starting.
I20260620 23:28:49.930897 5004 tablet_bootstrap.cc:654] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:49.932382 5004 log.cc:824] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:49.935293 5006 tablet_bootstrap.cc:492] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Bootstrap starting.
I20260620 23:28:49.935776 5004 tablet_bootstrap.cc:492] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: No bootstrap required, opened a new log
I20260620 23:28:49.936714 5007 tablet_bootstrap.cc:492] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Bootstrap starting.
I20260620 23:28:49.936852 5004 ts_tablet_manager.cc:1397] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 23:28:49.939481 5006 tablet_bootstrap.cc:654] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:49.940428 5007 tablet_bootstrap.cc:654] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:49.940819 5006 log.cc:824] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:49.942050 5007 log.cc:824] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:49.946004 5007 tablet_bootstrap.cc:492] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: No bootstrap required, opened a new log
I20260620 23:28:49.946429 5007 ts_tablet_manager.cc:1397] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260620 23:28:49.947154 5006 tablet_bootstrap.cc:492] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: No bootstrap required, opened a new log
I20260620 23:28:49.947515 5006 ts_tablet_manager.cc:1397] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260620 23:28:49.947728 5004 raft_consensus.cc:348] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.948220 5004 raft_consensus.cc:374] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:49.948323 5004 raft_consensus.cc:729] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e1134f95377494c96c73691b7649d5d, State: Initialized, Role: FOLLOWER
I20260620 23:28:49.949015 5004 consensus_queue.cc:260] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.950474 4853 heartbeater.cc:503] Master 127.0.96.254:42487 was elected leader, sending a full tablet report...
I20260620 23:28:49.951033 5004 ts_tablet_manager.cc:1428] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260620 23:28:49.957064 5007 raft_consensus.cc:348] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.957579 5007 raft_consensus.cc:374] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:49.957679 5007 raft_consensus.cc:729] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c542daa830124fca86afc976060444a7, State: Initialized, Role: FOLLOWER
I20260620 23:28:49.959126 5006 raft_consensus.cc:348] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.959694 5006 raft_consensus.cc:374] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:49.959517 5007 consensus_queue.cc:260] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.959796 5006 raft_consensus.cc:729] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a79287431d884dda9f121833fadf37c2, State: Initialized, Role: FOLLOWER
I20260620 23:28:49.960685 5006 consensus_queue.cc:260] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.961529 5007 ts_tablet_manager.cc:1428] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 23:28:49.963171 4986 heartbeater.cc:503] Master 127.0.96.254:42487 was elected leader, sending a full tablet report...
I20260620 23:28:49.964022 5006 ts_tablet_manager.cc:1428] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260620 23:28:49.980957 5012 raft_consensus.cc:482] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:49.981419 5012 raft_consensus.cc:504] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.983362 5012 leader_election.cc:288] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e1134f95377494c96c73691b7649d5d (127.0.96.194:44281), c542daa830124fca86afc976060444a7 (127.0.96.193:40721)
I20260620 23:28:49.991501 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.984160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:41143 (local address 127.0.96.193:40721)
0620 23:28:49.984384 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.984392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.984730 (+ 338us) server_negotiation.cc:407] Connection header received
0620 23:28:49.984820 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.984825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.984905 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.985007 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:49.985662 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.986925 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.989327 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.990585 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.990800 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.990897 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.991093 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.991249 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:28:49.991556 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.983828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40721 (local address 127.0.96.195:41143)
0620 23:28:49.984337 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.984366 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.984577 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.985225 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.985231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.985274 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:49.985494 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.985508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.987077 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.987080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.989107 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.989122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.990772 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.990776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.990921 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.990948 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.991038 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.991194 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328}
I20260620 23:28:49.992018 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.983875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:44281 (local address 127.0.96.195:59529)
0620 23:28:49.984967 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:49.984991 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:49.985226 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:49.985673 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:49.985677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:49.985702 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:28:49.986076 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.986095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.987556 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.987561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.989499 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:49.989508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.991249 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.991255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:49.991424 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.991450 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:28:49.991588 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:28:49.991717 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":625,"threads_started":1}
I20260620 23:28:49.992050 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:49.984113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:59529 (local address 127.0.96.194:44281)
0620 23:28:49.984349 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:49.984357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:49.985184 (+ 827us) server_negotiation.cc:407] Connection header received
0620 23:28:49.985393 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:49.985398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:49.985479 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:49.985602 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:28:49.986303 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.987373 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.989670 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:49.991000 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:49.991201 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:49.991270 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:49.991618 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 23:28:49.991791 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:28:49.993254 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fa31c33b99441089ce76673100ed7c" candidate_uuid: "a79287431d884dda9f121833fadf37c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1134f95377494c96c73691b7649d5d" is_pre_election: true
I20260620 23:28:49.993252 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fa31c33b99441089ce76673100ed7c" candidate_uuid: "a79287431d884dda9f121833fadf37c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c542daa830124fca86afc976060444a7" is_pre_election: true
I20260620 23:28:49.993853 4808 raft_consensus.cc:2454] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a79287431d884dda9f121833fadf37c2 in term 0.
I20260620 23:28:49.993853 4675 raft_consensus.cc:2454] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a79287431d884dda9f121833fadf37c2 in term 0.
I20260620 23:28:49.994719 4874 leader_election.cc:302] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e1134f95377494c96c73691b7649d5d, a79287431d884dda9f121833fadf37c2, c542daa830124fca86afc976060444a7; no voters:
I20260620 23:28:49.995255 5012 raft_consensus.cc:2781] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:49.995483 5012 raft_consensus.cc:482] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:49.995599 5012 raft_consensus.cc:3037] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:49.998342 5012 raft_consensus.cc:504] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:49.999293 5012 leader_election.cc:288] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [CANDIDATE]: Term 1 election: Requested vote from peers 0e1134f95377494c96c73691b7649d5d (127.0.96.194:44281), c542daa830124fca86afc976060444a7 (127.0.96.193:40721)
I20260620 23:28:49.999982 4808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fa31c33b99441089ce76673100ed7c" candidate_uuid: "a79287431d884dda9f121833fadf37c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e1134f95377494c96c73691b7649d5d"
I20260620 23:28:50.000023 4675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fa31c33b99441089ce76673100ed7c" candidate_uuid: "a79287431d884dda9f121833fadf37c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c542daa830124fca86afc976060444a7"
I20260620 23:28:50.000250 4808 raft_consensus.cc:3037] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:50.000298 4675 raft_consensus.cc:3037] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:50.002758 4808 raft_consensus.cc:2454] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a79287431d884dda9f121833fadf37c2 in term 1.
I20260620 23:28:50.003374 4877 leader_election.cc:302] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e1134f95377494c96c73691b7649d5d, a79287431d884dda9f121833fadf37c2; no voters:
I20260620 23:28:50.003409 4675 raft_consensus.cc:2454] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a79287431d884dda9f121833fadf37c2 in term 1.
I20260620 23:28:50.003906 5012 raft_consensus.cc:2781] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:50.004854 5012 raft_consensus.cc:686] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 1 LEADER]: Becoming Leader. State: Replica: a79287431d884dda9f121833fadf37c2, State: Running, Role: LEADER
I20260620 23:28:50.005734 5012 consensus_queue.cc:237] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } }
I20260620 23:28:50.012468 4532 catalog_manager.cc:5495] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a79287431d884dda9f121833fadf37c2 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "a79287431d884dda9f121833fadf37c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a79287431d884dda9f121833fadf37c2" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 } health_report { overall_health: UNKNOWN } } }
I20260620 23:28:50.046186 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:50.039913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.0.96.195:37667)
0620 23:28:50.040136 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:50.040141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:50.040281 (+ 140us) server_negotiation.cc:407] Connection header received
0620 23:28:50.040488 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:50.040491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:50.040581 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:50.040749 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:28:50.041338 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.042096 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.043792 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.044179 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.044280 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:50.044836 (+ 556us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:28:50.045499 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:28:50.045622 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:50.045801 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:28:50.045868 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 23:28:50.053067 4854 tablet.cc:2369] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:28:50.056064 4808 raft_consensus.cc:1261] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 1 FOLLOWER]: Refusing update from remote peer a79287431d884dda9f121833fadf37c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:50.056288 4675 raft_consensus.cc:1261] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 1 FOLLOWER]: Refusing update from remote peer a79287431d884dda9f121833fadf37c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:50.057406 5014 consensus_queue.cc:1035] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e1134f95377494c96c73691b7649d5d" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:50.057898 5012 consensus_queue.cc:1035] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c542daa830124fca86afc976060444a7" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:28:50.065092 4987 tablet.cc:2369] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:28:50.074373 5018 mvcc.cc:205] Tried to move back new op lower bound from 7299064340697137152 to 7299064340510273536. Current Snapshot: MvccSnapshot[applied={T|T < 7299064340697137152}]
I20260620 23:28:50.089818 4531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:50.090224 4531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:50.094741 4531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0906d1458f0c4454b276784c427f1bc5: Sending DeleteTablet for 3 replicas of tablet 25fa31c33b99441089ce76673100ed7c
I20260620 23:28:50.096418 4788 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fa31c33b99441089ce76673100ed7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:50 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 23:28:50.096426 4921 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fa31c33b99441089ce76673100ed7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:50 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:53552
I20260620 23:28:50.097666 4655 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fa31c33b99441089ce76673100ed7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:50 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:44774
I20260620 23:28:50.098546 5027 tablet_replica.cc:330] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: stopping tablet replica
I20260620 23:28:50.098881 5028 tablet_replica.cc:330] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: stopping tablet replica
I20260620 23:28:50.098955 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4593
I20260620 23:28:50.098945 5026 tablet_replica.cc:330] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: stopping tablet replica
I20260620 23:28:50.099478 5028 raft_consensus.cc:2229] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:50.099999 5027 raft_consensus.cc:2229] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:50.100147 5028 raft_consensus.cc:2258] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:50.100212 5026 raft_consensus.cc:2229] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:50.101011 5026 raft_consensus.cc:2258] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:50.101580 5027 raft_consensus.cc:2258] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:50.102979 5028 ts_tablet_manager.cc:1905] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:50.104012 5027 ts_tablet_manager.cc:1905] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:50.110826 5026 ts_tablet_manager.cc:1905] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:50.111035 5028 ts_tablet_manager.cc:1918] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:50.111255 5028 log.cc:1196] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/25fa31c33b99441089ce76673100ed7c
I20260620 23:28:50.111829 5028 ts_tablet_manager.cc:1939] T 25fa31c33b99441089ce76673100ed7c P c542daa830124fca86afc976060444a7: Deleting consensus metadata
I20260620 23:28:50.112068 5027 ts_tablet_manager.cc:1918] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:50.112262 5027 log.cc:1196] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/25fa31c33b99441089ce76673100ed7c
I20260620 23:28:50.112807 5027 ts_tablet_manager.cc:1939] T 25fa31c33b99441089ce76673100ed7c P 0e1134f95377494c96c73691b7649d5d: Deleting consensus metadata
I20260620 23:28:50.114382 4516 catalog_manager.cc:4841] TS c542daa830124fca86afc976060444a7 (127.0.96.193:40721): tablet 25fa31c33b99441089ce76673100ed7c (table test-table [id=03185523c29548139dcaf7d7e5046c46]) successfully deleted
I20260620 23:28:50.115731 4519 catalog_manager.cc:4841] TS 0e1134f95377494c96c73691b7649d5d (127.0.96.194:44281): tablet 25fa31c33b99441089ce76673100ed7c (table test-table [id=03185523c29548139dcaf7d7e5046c46]) successfully deleted
I20260620 23:28:50.121121 5026 ts_tablet_manager.cc:1918] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:50.121338 5026 log.cc:1196] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/25fa31c33b99441089ce76673100ed7c
I20260620 23:28:50.121904 5026 ts_tablet_manager.cc:1939] T 25fa31c33b99441089ce76673100ed7c P a79287431d884dda9f121833fadf37c2: Deleting consensus metadata
I20260620 23:28:50.123939 4517 catalog_manager.cc:4841] TS a79287431d884dda9f121833fadf37c2 (127.0.96.195:37667): tablet 25fa31c33b99441089ce76673100ed7c (table test-table [id=03185523c29548139dcaf7d7e5046c46]) successfully deleted
Jun 20 23:28:50 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:28:50.133635 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:50.102361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.96.193:40721)
0620 23:28:50.102632 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:50.102638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:50.115173 (+ 12535us) server_negotiation.cc:407] Connection header received
0620 23:28:50.115766 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:50.115773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:50.115896 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:50.115979 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:50.118021 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.118890 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.120270 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.120705 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.120851 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:50.126374 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:50.126395 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:50.126416 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:50.126477 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:50.130023 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:50.131131 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:50.131142 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:50.131144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:50.131375 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:50.131808 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:50.131813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:50.131817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:50.132252 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:50.132614 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:50.132906 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:28:50.133208 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 23:28:50.134513 4715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:50.260406 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4593
I20260620 23:28:50.281255 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4726
Jun 20 23:28:50 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:28:50.299955 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:50.282241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.96.194:44281)
0620 23:28:50.282482 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:50.282488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:50.282511 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:28:50.282725 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:50.282731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:50.282870 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:50.282994 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:50.284426 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.285505 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.286902 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.287355 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.287509 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:50.292588 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:50.292612 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:50.292634 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:50.292695 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:50.296291 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:50.297255 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:50.297264 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:50.297266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:50.297427 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:50.297831 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:50.297838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:50.297841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:50.298302 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:50.298641 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:50.299006 (+ 365us) server_negotiation.cc:299] Negotiation successful
0620 23:28:50.299267 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 23:28:50.300925 4848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:50.401158 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4726
I20260620 23:28:50.421725 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4859
Jun 20 23:28:50 dist-test-slave-v34r krb5kdc[4488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:28:50.442654 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:50.422706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.0.96.195:37667)
0620 23:28:50.422969 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:50.422976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:50.423001 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:28:50.423151 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:50.423157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:50.423260 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:50.423384 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:50.424739 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.425963 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.427310 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:50.428006 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:50.428166 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:50.434281 (+ 6115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:50.434301 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:50.434329 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:28:50.434383 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:50.438754 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:50.439778 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:50.439787 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:50.439791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:50.439978 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:28:50.440515 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:28:50.440521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:28:50.440524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:28:50.441182 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:50.441583 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:50.441937 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:28:50.442204 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 23:28:50.443552 4981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:50.589461 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4859
I20260620 23:28:50.613188 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 4501
I20260620 23:28:50.614611 4562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:50.737138 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 4501
2026-06-20T23:28:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4662 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T23:28:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:50Z Disabled control of system clock
I20260620 23:28:50.815160 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38061
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:28:51.149927 5037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:51.150240 5037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:51.150326 5037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:51.157590 5037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:51.157699 5037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:51.157737 5037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:51.157785 5037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:51.157821 5037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:28:51.167181 5037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:51.168469 5037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:51.170181 5037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:51.183286 5045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:51.183413 5043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:51.183570 5042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:51.184196 5037 server_base.cc:1034] running on GCE node
I20260620 23:28:51.185333 5037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:51.187110 5037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:51.188396 5037 hybrid_clock.cc:648] HybridClock initialized: now 1781998131188331 us; error 46 us; skew 500 ppm
I20260620 23:28:51.188925 5037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:51.192097 5037 webserver.cc:460] Webserver started at http://127.0.96.254:42245/ using document root <none> and password file <none>
I20260620 23:28:51.193039 5037 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:51.193163 5037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:51.193506 5037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:51.197151 5037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "ada87e8aa34d4c529e78837f46e222b2"
format_stamp: "Formatted at 2026-06-20 23:28:51 on dist-test-slave-v34r"
server_key: "0991ccd1bd06c88d29b9ebd101dd7bd1"
server_key_iv: "96cc73f4db4c6649ebc59c0db5790af7"
server_key_version: "encryptionkey@0"
I20260620 23:28:51.198264 5037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "ada87e8aa34d4c529e78837f46e222b2"
format_stamp: "Formatted at 2026-06-20 23:28:51 on dist-test-slave-v34r"
server_key: "0991ccd1bd06c88d29b9ebd101dd7bd1"
server_key_iv: "96cc73f4db4c6649ebc59c0db5790af7"
server_key_version: "encryptionkey@0"
I20260620 23:28:51.206159 5037 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260620 23:28:51.210877 5051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:51.212814 5037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:51.213133 5037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "ada87e8aa34d4c529e78837f46e222b2"
format_stamp: "Formatted at 2026-06-20 23:28:51 on dist-test-slave-v34r"
server_key: "0991ccd1bd06c88d29b9ebd101dd7bd1"
server_key_iv: "96cc73f4db4c6649ebc59c0db5790af7"
server_key_version: "encryptionkey@0"
I20260620 23:28:51.213441 5037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:51.220856 5037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:51.221979 5037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:51.222357 5037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:51.243526 5037 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38061
I20260620 23:28:51.243580 5102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38061 every 8 connection(s)
I20260620 23:28:51.245416 5037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:51.246305 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5037
I20260620 23:28:51.246515 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:51.247344 387 external_mini_cluster.cc:1428] Setting key 23bbe6fb972ce2a70393c1fb2bf751fb
I20260620 23:28:51.250872 5103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:51.258442 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:51.249907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.0.96.254:38061)
0620 23:28:51.251334 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:51.251350 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:51.251410 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:28:51.253419 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:51.253453 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:51.254113 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:51.254704 (+ 591us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:51.256138 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:51.256146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:51.256166 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:51.256187 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:51.256292 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:51.256315 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:51.256649 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:51.256854 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 23:28:51.257257 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":464,"threads_started":1}
I20260620 23:28:51.264170 5103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2: Bootstrap starting.
I20260620 23:28:51.267555 5103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:51.268669 5103 log.cc:824] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:51.272245 5103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2: No bootstrap required, opened a new log
I20260620 23:28:51.280019 5103 raft_consensus.cc:348] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada87e8aa34d4c529e78837f46e222b2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38061 } }
I20260620 23:28:51.280407 5103 raft_consensus.cc:374] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:51.280529 5103 raft_consensus.cc:729] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada87e8aa34d4c529e78837f46e222b2, State: Initialized, Role: FOLLOWER
I20260620 23:28:51.281206 5103 consensus_queue.cc:260] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada87e8aa34d4c529e78837f46e222b2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38061 } }
I20260620 23:28:51.281451 5103 raft_consensus.cc:388] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:51.281592 5103 raft_consensus.cc:482] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:51.281764 5103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:51.284089 5103 raft_consensus.cc:504] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada87e8aa34d4c529e78837f46e222b2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38061 } }
I20260620 23:28:51.284699 5103 leader_election.cc:302] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ada87e8aa34d4c529e78837f46e222b2; no voters:
I20260620 23:28:51.285460 5103 leader_election.cc:288] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:51.285691 5108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:51.286613 5108 raft_consensus.cc:686] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [term 1 LEADER]: Becoming Leader. State: Replica: ada87e8aa34d4c529e78837f46e222b2, State: Running, Role: LEADER
I20260620 23:28:51.287348 5108 consensus_queue.cc:237] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada87e8aa34d4c529e78837f46e222b2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38061 } }
I20260620 23:28:51.287673 5103 sys_catalog.cc:564] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:51.291074 5110 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ada87e8aa34d4c529e78837f46e222b2. Latest consensus state: current_term: 1 leader_uuid: "ada87e8aa34d4c529e78837f46e222b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada87e8aa34d4c529e78837f46e222b2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38061 } } }
I20260620 23:28:51.291847 5110 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:51.291975 5109 sys_catalog.cc:455] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ada87e8aa34d4c529e78837f46e222b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ada87e8aa34d4c529e78837f46e222b2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38061 } } }
I20260620 23:28:51.292418 5109 sys_catalog.cc:458] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:51.294181 5115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:51.301095 5115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:51.312644 5115 catalog_manager.cc:1347] Generated new cluster ID: 539a486e0c204e249887c9dd77e02dc2
I20260620 23:28:51.312803 5115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:51.333664 5115 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:51.334348 5115 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:51.348593 5115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2: Generated new TSK 0
I20260620 23:28:51.350099 5115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:28:51.361954 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38061
--builtin_ntp_servers=127.0.96.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 23:28:51.691778 5127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:51.692062 5127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:51.692183 5127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:51.699970 5127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:51.700073 5127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:51.700237 5127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:28:51.709251 5127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:51.710573 5127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:51.712280 5127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:51.726938 5133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:51.726958 5135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:51.727574 5132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:51.730716 5127 server_base.cc:1034] running on GCE node
I20260620 23:28:51.731560 5127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:51.732757 5127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:51.733989 5127 hybrid_clock.cc:648] HybridClock initialized: now 1781998131733898 us; error 80 us; skew 500 ppm
I20260620 23:28:51.734398 5127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:51.737380 5127 webserver.cc:460] Webserver started at http://127.0.96.193:40307/ using document root <none> and password file <none>
I20260620 23:28:51.738301 5127 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:51.738426 5127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:51.738853 5127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:51.742501 5127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "ddf1906c38b64833bc7110e472607750"
format_stamp: "Formatted at 2026-06-20 23:28:51 on dist-test-slave-v34r"
server_key: "33971678c0884a8effdb33807f743613"
server_key_iv: "2b06f2d95be144ad5a7145cbfd5f1aa9"
server_key_version: "encryptionkey@0"
I20260620 23:28:51.743484 5127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "ddf1906c38b64833bc7110e472607750"
format_stamp: "Formatted at 2026-06-20 23:28:51 on dist-test-slave-v34r"
server_key: "33971678c0884a8effdb33807f743613"
server_key_iv: "2b06f2d95be144ad5a7145cbfd5f1aa9"
server_key_version: "encryptionkey@0"
I20260620 23:28:51.750048 5127 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20260620 23:28:51.754547 5141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:51.756428 5127 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260620 23:28:51.756678 5127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "ddf1906c38b64833bc7110e472607750"
format_stamp: "Formatted at 2026-06-20 23:28:51 on dist-test-slave-v34r"
server_key: "33971678c0884a8effdb33807f743613"
server_key_iv: "2b06f2d95be144ad5a7145cbfd5f1aa9"
server_key_version: "encryptionkey@0"
I20260620 23:28:51.757053 5127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:51.764237 5127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:51.765403 5127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:51.765735 5127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:51.766932 5127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:51.768767 5127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:51.768914 5127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:51.769099 5127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:51.769193 5127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:51.808820 5127 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:41719
I20260620 23:28:51.808838 5253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:41719 every 8 connection(s)
I20260620 23:28:51.810559 5127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:28:51.813448 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5127
I20260620 23:28:51.813686 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:28:51.814586 387 external_mini_cluster.cc:1428] Setting key 19bd3c52eaa260a4d5f119aa555e1c39
I20260620 23:28:51.818503 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38061
--builtin_ntp_servers=127.0.96.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:28:51.821491 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:51.813954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52219 (local address 127.0.96.254:38061)
0620 23:28:51.814413 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:51.814418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:51.816743 (+ 2325us) server_negotiation.cc:407] Connection header received
0620 23:28:51.819031 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:51.819036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:51.819104 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:51.819237 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:51.820647 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:51.820653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:51.820658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:51.820677 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:51.820731 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:51.820743 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:51.820880 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:51.821114 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 23:28:51.821181 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":257,"threads_started":1}
I20260620 23:28:51.822439 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:51.814961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38061 (local address 127.0.96.193:52219)
0620 23:28:51.816530 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:51.816596 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:51.818692 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:51.819497 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:51.819506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:51.819607 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:51.820351 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:51.820354 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:51.820395 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:51.820484 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:51.820839 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:51.820849 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:51.820857 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:28:51.821192 (+ 335us) client_negotiation.cc:240] Negotiation successful
0620 23:28:51.821464 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":526,"threads_started":1}
I20260620 23:28:51.824224 5254 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38061
I20260620 23:28:51.824542 5254 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:51.825304 5254 heartbeater.cc:511] Master 127.0.96.254:38061 requested a full tablet report, sending...
I20260620 23:28:51.828141 5068 ts_manager.cc:194] Registered new tserver with Master: ddf1906c38b64833bc7110e472607750 (127.0.96.193:41719)
W20260620 23:28:52.153647 5258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:52.154001 5258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:52.154153 5258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:52.161495 5258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:52.161659 5258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:52.161842 5258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:28:52.171002 5258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:52.172353 5258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:52.174124 5258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:52.189656 5266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:52.189715 5264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:52.189612 5263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:52.192031 5258 server_base.cc:1034] running on GCE node
I20260620 23:28:52.192770 5258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:52.193969 5258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:52.195254 5258 hybrid_clock.cc:648] HybridClock initialized: now 1781998132195196 us; error 55 us; skew 500 ppm
I20260620 23:28:52.195712 5258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:52.198427 5258 webserver.cc:460] Webserver started at http://127.0.96.194:37727/ using document root <none> and password file <none>
I20260620 23:28:52.199193 5258 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:52.199328 5258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:52.199658 5258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:52.202878 5258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "815e1e726980479183d30a4ecdda4909"
format_stamp: "Formatted at 2026-06-20 23:28:52 on dist-test-slave-v34r"
server_key: "9ebbee6d67723e9dac7d07c9358a8594"
server_key_iv: "0aac88e71f96d54fc96901331e6a6e28"
server_key_version: "encryptionkey@0"
I20260620 23:28:52.203747 5258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "815e1e726980479183d30a4ecdda4909"
format_stamp: "Formatted at 2026-06-20 23:28:52 on dist-test-slave-v34r"
server_key: "9ebbee6d67723e9dac7d07c9358a8594"
server_key_iv: "0aac88e71f96d54fc96901331e6a6e28"
server_key_version: "encryptionkey@0"
I20260620 23:28:52.209661 5258 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:28:52.213846 5272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:52.215529 5258 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:28:52.215747 5258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "815e1e726980479183d30a4ecdda4909"
format_stamp: "Formatted at 2026-06-20 23:28:52 on dist-test-slave-v34r"
server_key: "9ebbee6d67723e9dac7d07c9358a8594"
server_key_iv: "0aac88e71f96d54fc96901331e6a6e28"
server_key_version: "encryptionkey@0"
I20260620 23:28:52.216024 5258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:52.222324 5258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:52.223184 5258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:52.223475 5258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:52.224467 5258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:52.226081 5258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:52.226212 5258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:52.226356 5258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:52.226446 5258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:52.276053 5258 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:36993
I20260620 23:28:52.276157 5384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:36993 every 8 connection(s)
I20260620 23:28:52.277581 5258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:28:52.284329 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5258
I20260620 23:28:52.284660 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:28:52.285311 387 external_mini_cluster.cc:1428] Setting key b491c4474d5814b786572de31fa0afbe
I20260620 23:28:52.288385 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.281208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:58987 (local address 127.0.96.254:38061)
0620 23:28:52.281431 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:52.281438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:52.283190 (+ 1752us) server_negotiation.cc:407] Connection header received
0620 23:28:52.285423 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:52.285430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:52.285522 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:52.285658 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:52.287337 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:52.287350 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:52.287354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:52.287376 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:52.287460 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:52.287483 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:52.287633 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:52.287974 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:28:52.288054 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:28:52.289310 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38061
--builtin_ntp_servers=127.0.96.212:44963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 23:28:52.289739 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.281814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38061 (local address 127.0.96.194:58987)
0620 23:28:52.282985 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:52.283037 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:52.285106 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:52.285931 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:52.285941 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:52.286052 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:52.286903 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:52.286907 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:52.286974 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:52.287097 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:52.287626 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:52.287640 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:52.287649 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:28:52.288043 (+ 394us) client_negotiation.cc:240] Negotiation successful
0620 23:28:52.288309 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":339,"threads_started":1}
I20260620 23:28:52.291625 5385 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38061
I20260620 23:28:52.292109 5385 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:52.293093 5385 heartbeater.cc:511] Master 127.0.96.254:38061 requested a full tablet report, sending...
I20260620 23:28:52.295156 5068 ts_manager.cc:194] Registered new tserver with Master: 815e1e726980479183d30a4ecdda4909 (127.0.96.194:36993)
W20260620 23:28:52.620775 5388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:52.621097 5388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:52.621243 5388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:52.628640 5388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:52.628758 5388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:52.628921 5388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:28:52.638207 5388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:52.639554 5388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:52.641356 5388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:52.655946 5393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:52.656589 5396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:52.657629 5394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:52.658833 5388 server_base.cc:1034] running on GCE node
I20260620 23:28:52.659581 5388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:52.660682 5388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:52.662047 5388 hybrid_clock.cc:648] HybridClock initialized: now 1781998132661950 us; error 64 us; skew 500 ppm
I20260620 23:28:52.662469 5388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:52.665412 5388 webserver.cc:460] Webserver started at http://127.0.96.195:40047/ using document root <none> and password file <none>
I20260620 23:28:52.666209 5388 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:52.666350 5388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:52.666694 5388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:52.669840 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "1eb71df3353149068ba38a39ef27e625"
format_stamp: "Formatted at 2026-06-20 23:28:52 on dist-test-slave-v34r"
server_key: "c61b68c627e05ae07a0314b0ad76b933"
server_key_iv: "99a335b4e2f9a524cc842476a29d6480"
server_key_version: "encryptionkey@0"
I20260620 23:28:52.670725 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "1eb71df3353149068ba38a39ef27e625"
format_stamp: "Formatted at 2026-06-20 23:28:52 on dist-test-slave-v34r"
server_key: "c61b68c627e05ae07a0314b0ad76b933"
server_key_iv: "99a335b4e2f9a524cc842476a29d6480"
server_key_version: "encryptionkey@0"
I20260620 23:28:52.676442 5388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 23:28:52.680552 5402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:52.682662 5388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 23:28:52.683034 5388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "1eb71df3353149068ba38a39ef27e625"
format_stamp: "Formatted at 2026-06-20 23:28:52 on dist-test-slave-v34r"
server_key: "c61b68c627e05ae07a0314b0ad76b933"
server_key_iv: "99a335b4e2f9a524cc842476a29d6480"
server_key_version: "encryptionkey@0"
I20260620 23:28:52.683343 5388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:52.689883 5388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:52.690752 5388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:52.691030 5388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:52.692054 5388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:28:52.693640 5388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:28:52.693760 5388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:52.693897 5388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:28:52.693984 5388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:52.732910 5388 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:43813
I20260620 23:28:52.733093 5514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:43813 every 8 connection(s)
I20260620 23:28:52.734242 5388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:28:52.735704 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5388
I20260620 23:28:52.735881 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:28:52.736378 387 external_mini_cluster.cc:1428] Setting key ec3142ec0dca70ca50293e9a875c9319
I20260620 23:28:52.744091 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.737546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:49223 (local address 127.0.96.254:38061)
0620 23:28:52.737773 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:52.737786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:52.740050 (+ 2264us) server_negotiation.cc:407] Connection header received
0620 23:28:52.741799 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:52.741803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:52.741887 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:52.741994 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:52.743371 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:52.743375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:52.743379 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:52.743397 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:52.743437 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:52.743450 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:52.743574 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:52.743754 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 23:28:52.743818 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:28:52.745041 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.738377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38061 (local address 127.0.96.195:49223)
0620 23:28:52.739848 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:52.739908 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:52.741527 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:52.742203 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:52.742212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:52.742303 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:52.743037 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:52.743040 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:52.743076 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:52.743160 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:52.743562 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:52.743568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:52.743574 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:28:52.743800 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 23:28:52.743996 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":401,"threads_started":1}
I20260620 23:28:52.746903 5515 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38061
I20260620 23:28:52.747268 5515 heartbeater.cc:464] Registering TS with master...
I20260620 23:28:52.747967 5515 heartbeater.cc:511] Master 127.0.96.254:38061 requested a full tablet report, sending...
I20260620 23:28:52.749709 5068 ts_manager.cc:194] Registered new tserver with Master: 1eb71df3353149068ba38a39ef27e625 (127.0.96.195:43813)
I20260620 23:28:52.752717 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:28:52.760726 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.757960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.96.254:38061)
0620 23:28:52.758209 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:52.758215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:52.758661 (+ 446us) server_negotiation.cc:407] Connection header received
0620 23:28:52.758823 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:52.758827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:52.758893 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:52.759019 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:52.760048 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:52.760052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:52.760056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:52.760072 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:52.760109 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:52.760123 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:52.760251 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:52.760400 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 23:28:52.760457 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:28:52.766335 5068 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:28:52.774530 5068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:28:52.808183 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.790143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.0.96.193:41719)
0620 23:28:52.805219 (+ 15076us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:52.805227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:52.805260 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:28:52.805333 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:52.805339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:52.805555 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:52.805725 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:52.807201 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:52.807212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:52.807229 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:52.807249 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:52.807291 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:52.807306 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:52.807450 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:52.807644 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 23:28:52.807744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14941,"thread_start_us":14896,"threads_started":1}
I20260620 23:28:52.812206 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.790058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37726)
0620 23:28:52.799233 (+ 9175us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:52.799303 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:52.799558 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:52.805751 (+ 6193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:52.805758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:52.805856 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:52.806933 (+ 1077us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:52.806935 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:52.806966 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:52.807044 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:52.807404 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:52.807414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:52.807421 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:28:52.811510 (+ 4089us) client_negotiation.cc:240] Negotiation successful
0620 23:28:52.811648 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8730,"thread_start_us":14295,"threads_started":1}
I20260620 23:28:52.813084 5527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.790649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:36993 (local address 127.0.0.1:45372)
0620 23:28:52.804671 (+ 14022us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:52.804688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:52.804783 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:52.809045 (+ 4262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:52.809052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:52.809081 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:52.809950 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:52.809954 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:52.810079 (+ 125us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:52.811734 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:52.812322 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:52.812328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:52.812337 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 23:28:52.812506 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 23:28:52.812569 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13873,"thread_start_us":19200,"threads_started":1}
I20260620 23:28:52.813553 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.790940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.0.96.194:36993)
0620 23:28:52.808119 (+ 17179us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:52.808135 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:52.808191 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:28:52.808329 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:52.808342 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:52.808757 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:52.810238 (+ 1481us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:52.811907 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:52.811920 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:52.811937 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:52.811956 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:52.812038 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:52.812069 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:52.812258 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:52.813046 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 23:28:52.813195 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16939,"thread_start_us":18086,"threads_started":1}
I20260620 23:28:52.814153 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.796250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46694 (local address 127.0.96.195:43813)
0620 23:28:52.800712 (+ 4462us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:52.800726 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:52.810025 (+ 9299us) server_negotiation.cc:407] Connection header received
0620 23:28:52.810187 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:52.810200 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:52.810451 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:52.810627 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:52.813169 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:52.813179 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:52.813195 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:52.813214 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:52.813277 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:52.813295 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:52.813434 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:52.813693 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 23:28:52.813804 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4194,"thread_start_us":3951,"threads_started":1}
I20260620 23:28:52.815037 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:52.795318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:43813 (local address 127.0.0.1:46694)
0620 23:28:52.809867 (+ 14549us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:52.809897 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:52.810050 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:52.811900 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:52.811907 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:52.811943 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:52.812888 (+ 945us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:52.812892 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:52.812912 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:52.812963 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:52.813533 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:52.813539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:52.813543 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 23:28:52.814481 (+ 938us) client_negotiation.cc:240] Negotiation successful
0620 23:28:52.814560 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14337,"thread_start_us":3175,"threads_started":1}
I20260620 23:28:52.818463 5320 tablet_service.cc:1467] Processing CreateTablet for tablet 1f52519ef3284ce683af627dd9839500 (DEFAULT_TABLE table=test-table [id=a92487bf9b834742b5a8978269f329e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:52.818831 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 1f52519ef3284ce683af627dd9839500 (DEFAULT_TABLE table=test-table [id=a92487bf9b834742b5a8978269f329e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:52.820017 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f52519ef3284ce683af627dd9839500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:52.819469 5450 tablet_service.cc:1467] Processing CreateTablet for tablet 1f52519ef3284ce683af627dd9839500 (DEFAULT_TABLE table=test-table [id=a92487bf9b834742b5a8978269f329e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:28:52.820056 5320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f52519ef3284ce683af627dd9839500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:52.821090 5450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f52519ef3284ce683af627dd9839500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:52.832048 5254 heartbeater.cc:503] Master 127.0.96.254:38061 was elected leader, sending a full tablet report...
I20260620 23:28:52.840374 5533 tablet_bootstrap.cc:492] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Bootstrap starting.
I20260620 23:28:52.845839 5534 tablet_bootstrap.cc:492] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: Bootstrap starting.
I20260620 23:28:52.848793 5533 tablet_bootstrap.cc:654] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:52.849421 5534 tablet_bootstrap.cc:654] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:52.850466 5534 log.cc:824] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:52.850855 5533 log.cc:824] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:52.851730 5535 tablet_bootstrap.cc:492] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Bootstrap starting.
I20260620 23:28:52.855149 5533 tablet_bootstrap.cc:492] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: No bootstrap required, opened a new log
I20260620 23:28:52.855150 5534 tablet_bootstrap.cc:492] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: No bootstrap required, opened a new log
I20260620 23:28:52.855225 5535 tablet_bootstrap.cc:654] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:52.855536 5533 ts_tablet_manager.cc:1397] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260620 23:28:52.856220 5534 ts_tablet_manager.cc:1397] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:28:52.856417 5535 log.cc:824] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:52.859144 5535 tablet_bootstrap.cc:492] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: No bootstrap required, opened a new log
I20260620 23:28:52.859418 5535 ts_tablet_manager.cc:1397] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 23:28:52.866863 5535 raft_consensus.cc:348] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:52.867316 5535 raft_consensus.cc:374] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:52.867419 5535 raft_consensus.cc:729] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb71df3353149068ba38a39ef27e625, State: Initialized, Role: FOLLOWER
I20260620 23:28:52.867220 5533 raft_consensus.cc:348] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:52.867781 5533 raft_consensus.cc:374] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:52.867888 5533 raft_consensus.cc:729] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815e1e726980479183d30a4ecdda4909, State: Initialized, Role: FOLLOWER
I20260620 23:28:52.867828 5534 raft_consensus.cc:348] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:52.868271 5534 raft_consensus.cc:374] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:52.868124 5535 consensus_queue.cc:260] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:52.868357 5534 raft_consensus.cc:729] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddf1906c38b64833bc7110e472607750, State: Initialized, Role: FOLLOWER
I20260620 23:28:52.868629 5533 consensus_queue.cc:260] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:52.868952 5534 consensus_queue.cc:260] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:52.870203 5385 heartbeater.cc:503] Master 127.0.96.254:38061 was elected leader, sending a full tablet report...
I20260620 23:28:52.871013 5533 ts_tablet_manager.cc:1428] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 23:28:52.873564 5534 ts_tablet_manager.cc:1428] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260620 23:28:52.876937 5535 ts_tablet_manager.cc:1428] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Time spent starting tablet: real 0.017s user 0.001s sys 0.017s
I20260620 23:28:52.877091 5515 heartbeater.cc:503] Master 127.0.96.254:38061 was elected leader, sending a full tablet report...
W20260620 23:28:52.986280 5516 tablet.cc:2369] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:28:53.030081 5386 tablet.cc:2369] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:28:53.064452 5255 tablet.cc:2369] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:28:53.135094 5541 raft_consensus.cc:482] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:28:53.135412 5541 raft_consensus.cc:504] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:53.137344 5541 leader_election.cc:288] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 815e1e726980479183d30a4ecdda4909 (127.0.96.194:36993), 1eb71df3353149068ba38a39ef27e625 (127.0.96.195:43813)
I20260620 23:28:53.141544 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.137907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:51831 (local address 127.0.96.194:36993)
0620 23:28:53.138177 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:53.138186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:53.139071 (+ 885us) server_negotiation.cc:407] Connection header received
0620 23:28:53.139336 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:53.139341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:53.139404 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:53.139522 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:53.140765 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:53.140771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:53.140777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:53.140802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:53.140838 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:53.140851 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:53.141018 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:53.141113 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 23:28:53.141200 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:28:53.141641 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.138057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52311 (local address 127.0.96.195:43813)
0620 23:28:53.138406 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:53.138415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:53.139256 (+ 841us) server_negotiation.cc:407] Connection header received
0620 23:28:53.139493 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:53.139500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:53.139554 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:53.139662 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:53.140930 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:53.140936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:53.140951 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:53.140968 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:53.141047 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:53.141069 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:53.141214 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:53.141308 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 23:28:53.141374 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260620 23:28:53.141961 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.137680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:36993 (local address 127.0.96.193:51831)
0620 23:28:53.138883 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.138912 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:53.139119 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:53.139551 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:53.139557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:53.139584 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:53.140493 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:53.140499 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:53.140522 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:53.140584 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:53.140955 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:53.140960 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:53.140965 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 23:28:53.141165 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 23:28:53.141223 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":637,"threads_started":1}
I20260620 23:28:53.142349 5543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.138055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:43813 (local address 127.0.96.193:52311)
0620 23:28:53.139106 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.139135 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:53.139346 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:53.139702 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:53.139708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:53.139736 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:28:53.140686 (+ 950us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:28:53.140689 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:28:53.140712 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:28:53.140754 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:28:53.141173 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:28:53.141179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:28:53.141185 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:28:53.141358 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 23:28:53.141420 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":643,"threads_started":1}
I20260620 23:28:53.143339 5340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f52519ef3284ce683af627dd9839500" candidate_uuid: "ddf1906c38b64833bc7110e472607750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "815e1e726980479183d30a4ecdda4909" is_pre_election: true
I20260620 23:28:53.143540 5470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f52519ef3284ce683af627dd9839500" candidate_uuid: "ddf1906c38b64833bc7110e472607750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb71df3353149068ba38a39ef27e625" is_pre_election: true
I20260620 23:28:53.143931 5340 raft_consensus.cc:2454] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf1906c38b64833bc7110e472607750 in term 0.
I20260620 23:28:53.144222 5470 raft_consensus.cc:2454] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddf1906c38b64833bc7110e472607750 in term 0.
I20260620 23:28:53.144726 5144 leader_election.cc:302] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 815e1e726980479183d30a4ecdda4909, ddf1906c38b64833bc7110e472607750; no voters:
I20260620 23:28:53.145258 5541 raft_consensus.cc:2781] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:28:53.145406 5541 raft_consensus.cc:482] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:28:53.145524 5541 raft_consensus.cc:3037] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:53.147622 5541 raft_consensus.cc:504] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:53.148252 5541 leader_election.cc:288] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [CANDIDATE]: Term 1 election: Requested vote from peers 815e1e726980479183d30a4ecdda4909 (127.0.96.194:36993), 1eb71df3353149068ba38a39ef27e625 (127.0.96.195:43813)
I20260620 23:28:53.148837 5340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f52519ef3284ce683af627dd9839500" candidate_uuid: "ddf1906c38b64833bc7110e472607750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "815e1e726980479183d30a4ecdda4909"
I20260620 23:28:53.149078 5340 raft_consensus.cc:3037] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:53.148969 5470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f52519ef3284ce683af627dd9839500" candidate_uuid: "ddf1906c38b64833bc7110e472607750" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eb71df3353149068ba38a39ef27e625"
I20260620 23:28:53.149323 5470 raft_consensus.cc:3037] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:53.151373 5340 raft_consensus.cc:2454] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf1906c38b64833bc7110e472607750 in term 1.
I20260620 23:28:53.151834 5144 leader_election.cc:302] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 815e1e726980479183d30a4ecdda4909, ddf1906c38b64833bc7110e472607750; no voters:
I20260620 23:28:53.151952 5470 raft_consensus.cc:2454] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddf1906c38b64833bc7110e472607750 in term 1.
I20260620 23:28:53.152125 5541 raft_consensus.cc:2781] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:53.152758 5541 raft_consensus.cc:686] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 1 LEADER]: Becoming Leader. State: Replica: ddf1906c38b64833bc7110e472607750, State: Running, Role: LEADER
I20260620 23:28:53.153421 5541 consensus_queue.cc:237] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } }
I20260620 23:28:53.159366 5068 catalog_manager.cc:5495] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddf1906c38b64833bc7110e472607750 (127.0.96.193). New cstate: current_term: 1 leader_uuid: "ddf1906c38b64833bc7110e472607750" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddf1906c38b64833bc7110e472607750" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 } health_report { overall_health: UNKNOWN } } }
I20260620 23:28:53.196735 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.194181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37738 (local address 127.0.96.193:41719)
0620 23:28:53.194372 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:53.194377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:53.194397 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:28:53.194529 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:53.194533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:53.194633 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:53.194757 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:53.195952 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:53.195958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:53.195973 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:53.195994 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:53.196046 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:53.196060 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:53.196187 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:53.196403 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 23:28:53.196474 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:28:53.207297 5340 raft_consensus.cc:1261] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 1 FOLLOWER]: Refusing update from remote peer ddf1906c38b64833bc7110e472607750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:53.207300 5470 raft_consensus.cc:1261] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 1 FOLLOWER]: Refusing update from remote peer ddf1906c38b64833bc7110e472607750: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:28:53.208724 5544 consensus_queue.cc:1035] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "815e1e726980479183d30a4ecdda4909" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:53.209261 5541 consensus_queue.cc:1035] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eb71df3353149068ba38a39ef27e625" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:28:53.225031 5548 mvcc.cc:205] Tried to move back new op lower bound from 7299064353604575232 to 7299064353400745984. Current Snapshot: MvccSnapshot[applied={T|T < 7299064353604575232}]
I20260620 23:28:53.225034 5549 mvcc.cc:205] Tried to move back new op lower bound from 7299064353604575232 to 7299064353400745984. Current Snapshot: MvccSnapshot[applied={T|T < 7299064353604575232}]
I20260620 23:28:53.238027 5067 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:53.238401 5067 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:28:53.243259 5067 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ada87e8aa34d4c529e78837f46e222b2: Sending DeleteTablet for 3 replicas of tablet 1f52519ef3284ce683af627dd9839500
I20260620 23:28:53.244704 5189 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f52519ef3284ce683af627dd9839500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:53 UTC) from {username='slave'} at 127.0.0.1:37726
I20260620 23:28:53.244891 5450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f52519ef3284ce683af627dd9839500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:53 UTC) from {username='slave'} at 127.0.0.1:46694
I20260620 23:28:53.246090 5320 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f52519ef3284ce683af627dd9839500 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:28:53 UTC) from {username='slave'} at 127.0.0.1:45372
I20260620 23:28:53.252259 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5127
I20260620 23:28:53.259104 5558 tablet_replica.cc:330] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: stopping tablet replica
I20260620 23:28:53.259896 5558 raft_consensus.cc:2229] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:53.260704 5558 raft_consensus.cc:2258] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:53.264518 5559 tablet_replica.cc:330] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: stopping tablet replica
I20260620 23:28:53.265131 5559 raft_consensus.cc:2229] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:28:53.265749 5559 raft_consensus.cc:2258] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:28:53.267225 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.254551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.0.96.193:41719)
0620 23:28:53.254840 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:53.254851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:53.262091 (+ 7240us) server_negotiation.cc:407] Connection header received
0620 23:28:53.262454 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:53.262460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:53.262546 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:53.262627 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:53.266257 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:53.266266 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:53.266272 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:53.266305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:53.266364 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:53.266389 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:53.266510 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:53.266786 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 23:28:53.266864 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:28:53.268332 5557 tablet_replica.cc:330] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750: stopping tablet replica
I20260620 23:28:53.268603 5249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:53.270017 5558 ts_tablet_manager.cc:1905] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:53.270849 5559 ts_tablet_manager.cc:1905] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:28:53.278621 5558 ts_tablet_manager.cc:1918] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:53.278864 5558 log.cc:1196] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/1f52519ef3284ce683af627dd9839500
I20260620 23:28:53.279628 5558 ts_tablet_manager.cc:1939] T 1f52519ef3284ce683af627dd9839500 P 1eb71df3353149068ba38a39ef27e625: Deleting consensus metadata
I20260620 23:28:53.282174 5053 catalog_manager.cc:4841] TS 1eb71df3353149068ba38a39ef27e625 (127.0.96.195:43813): tablet 1f52519ef3284ce683af627dd9839500 (table test-table [id=a92487bf9b834742b5a8978269f329e1]) successfully deleted
I20260620 23:28:53.289186 5559 ts_tablet_manager.cc:1918] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:28:53.289438 5559 log.cc:1196] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/1f52519ef3284ce683af627dd9839500
I20260620 23:28:53.289932 5559 ts_tablet_manager.cc:1939] T 1f52519ef3284ce683af627dd9839500 P 815e1e726980479183d30a4ecdda4909: Deleting consensus metadata
I20260620 23:28:53.291886 5054 catalog_manager.cc:4841] TS 815e1e726980479183d30a4ecdda4909 (127.0.96.194:36993): tablet 1f52519ef3284ce683af627dd9839500 (table test-table [id=a92487bf9b834742b5a8978269f329e1]) successfully deleted
I20260620 23:28:53.268913 5557 raft_consensus.cc:2229] T 1f52519ef3284ce683af627dd9839500 P ddf1906c38b64833bc7110e472607750 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:28:53.391486 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5127
W20260620 23:28:53.409701 5052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:41719 (error 108)
I20260620 23:28:53.410521 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5258
I20260620 23:28:53.411847 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.410967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37748)
0620 23:28:53.411501 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.411655 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":245,"threads_started":1}
W20260620 23:28:53.412173 5052 catalog_manager.cc:4584] TS ddf1906c38b64833bc7110e472607750 (127.0.96.193:41719): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f52519ef3284ce683af627dd9839500: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
I20260620 23:28:53.413865 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.411451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.0.96.194:36993)
0620 23:28:53.411626 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:53.411633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:53.411723 (+ 90us) server_negotiation.cc:407] Connection header received
0620 23:28:53.411944 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:53.411949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:53.412037 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:53.412144 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:53.413319 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:53.413323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:53.413326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:53.413341 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:53.413375 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:53.413387 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:53.413500 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:53.413583 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 23:28:53.413631 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:28:53.414538 5380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:53.466456 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.465655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37760)
0620 23:28:53.465999 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.466180 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 23:28:53.467924 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.467408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37776)
0620 23:28:53.467594 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.467705 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 23:28:53.515558 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5258
I20260620 23:28:53.528963 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.528142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37792)
0620 23:28:53.528534 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.528709 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 23:28:53.530412 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.529876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37808)
0620 23:28:53.530074 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.530190 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:28:53.534724 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5388
I20260620 23:28:53.537952 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.535632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46702 (local address 127.0.96.195:43813)
0620 23:28:53.535875 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:53.535882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:53.535905 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:28:53.536043 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:53.536048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:53.536161 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:53.536291 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:53.537270 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:53.537277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:53.537283 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:53.537305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:53.537343 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:53.537355 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:53.537507 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:53.537564 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 23:28:53.537631 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:28:53.538754 5510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:53.611362 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.610676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37816)
0620 23:28:53.610993 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.611166 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 23:28:53.612756 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:53.612241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41719 (local address 127.0.0.1:37822)
0620 23:28:53.612486 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:53.612592 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:41719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260620 23:28:53.627084 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5388
I20260620 23:28:53.643953 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5037
I20260620 23:28:53.645304 5098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:53.754371 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5037
2026-06-20T23:28:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3011 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 23:28:53.796159 387 mini_oidc.cc:152] Starting JWKS server
I20260620 23:28:53.798816 387 webserver.cc:460] Webserver started at https://127.0.0.1:33263/ using document root <none> and password file <none>
I20260620 23:28:53.799103 387 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:28:53.799800 387 webserver.cc:460] Webserver started at http://127.0.0.1:38083/ using document root <none> and password file <none>
2026-06-20T23:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:53Z Disabled control of system clock
I20260620 23:28:53.830255 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:45479
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:45957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:45479
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33263/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/testchainca.cert with env {}
W20260620 23:28:54.237365 5572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:54.237761 5572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:54.237923 5572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:54.245571 5572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:54.245738 5572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:54.245831 5572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:54.245939 5572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:54.246031 5572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:28:54.246120 5572 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:28:54.246209 5572 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33263/jwks.json
I20260620 23:28:54.256765 5572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:45479
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:45479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33263/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:54.258293 5572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:54.259644 5572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:54.273303 5578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:54.273574 5580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:54.273599 5577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:54.274941 5572 server_base.cc:1034] running on GCE node
I20260620 23:28:54.276163 5572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:54.278317 5572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:54.279620 5572 hybrid_clock.cc:648] HybridClock initialized: now 1781998134279562 us; error 53 us; skew 500 ppm
I20260620 23:28:54.280123 5572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:54.283262 5572 webserver.cc:460] Webserver started at http://127.0.96.254:45889/ using document root <none> and password file <none>
I20260620 23:28:54.284205 5572 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:54.284368 5572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:54.284722 5572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:54.288360 5572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "b78807d2f02c4d739be11ba0cd3af863"
format_stamp: "Formatted at 2026-06-20 23:28:54 on dist-test-slave-v34r"
server_key: "73069e14c98d77d56919e1d474129809"
server_key_iv: "097684bfc21c30afb55a737794f0dd74"
server_key_version: "encryptionkey@0"
I20260620 23:28:54.289324 5572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "b78807d2f02c4d739be11ba0cd3af863"
format_stamp: "Formatted at 2026-06-20 23:28:54 on dist-test-slave-v34r"
server_key: "73069e14c98d77d56919e1d474129809"
server_key_iv: "097684bfc21c30afb55a737794f0dd74"
server_key_version: "encryptionkey@0"
I20260620 23:28:54.297381 5572 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 23:28:54.302783 5586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:54.304898 5572 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:28:54.305274 5572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "b78807d2f02c4d739be11ba0cd3af863"
format_stamp: "Formatted at 2026-06-20 23:28:54 on dist-test-slave-v34r"
server_key: "73069e14c98d77d56919e1d474129809"
server_key_iv: "097684bfc21c30afb55a737794f0dd74"
server_key_version: "encryptionkey@0"
I20260620 23:28:54.305650 5572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:54.330466 5572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:54.331641 5572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:54.332069 5572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:54.353888 5572 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:45479
I20260620 23:28:54.353928 5637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:45479 every 8 connection(s)
I20260620 23:28:54.355651 5572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:54.360172 5638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:54.365339 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5572
I20260620 23:28:54.365618 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:54.366350 387 external_mini_cluster.cc:1428] Setting key 592cb43ee3a75dff4333cbfe5e38b223
I20260620 23:28:54.372364 5638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863: Bootstrap starting.
I20260620 23:28:54.376426 5638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:54.377568 5638 log.cc:824] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:54.381134 5638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863: No bootstrap required, opened a new log
I20260620 23:28:54.383805 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:54.368967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.0.96.254:45479)
0620 23:28:54.370462 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:54.370480 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:54.370541 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:28:54.372933 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:54.372964 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:54.373609 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:54.374114 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:54.374717 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:54.376656 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:54.378516 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:54.379390 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:54.379573 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:54.380895 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:54.380905 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:54.380923 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:54.381687 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:54.381793 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:54.381819 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:54.381961 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:54.382199 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 23:28:54.382530 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":455,"threads_started":1}
I20260620 23:28:54.388877 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:28:54.390489 5638 raft_consensus.cc:348] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78807d2f02c4d739be11ba0cd3af863" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45479 } }
I20260620 23:28:54.390831 5638 raft_consensus.cc:374] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:54.390954 5638 raft_consensus.cc:729] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b78807d2f02c4d739be11ba0cd3af863, State: Initialized, Role: FOLLOWER
I20260620 23:28:54.391595 5638 consensus_queue.cc:260] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78807d2f02c4d739be11ba0cd3af863" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45479 } }
I20260620 23:28:54.391862 5638 raft_consensus.cc:388] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:54.392021 5638 raft_consensus.cc:482] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:54.392232 5638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:54.394629 5638 raft_consensus.cc:504] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78807d2f02c4d739be11ba0cd3af863" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45479 } }
I20260620 23:28:54.395105 5638 leader_election.cc:302] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b78807d2f02c4d739be11ba0cd3af863; no voters:
I20260620 23:28:54.395700 5638 leader_election.cc:288] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:54.395979 5643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:54.396833 5643 raft_consensus.cc:686] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [term 1 LEADER]: Becoming Leader. State: Replica: b78807d2f02c4d739be11ba0cd3af863, State: Running, Role: LEADER
I20260620 23:28:54.397572 5643 consensus_queue.cc:237] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78807d2f02c4d739be11ba0cd3af863" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45479 } }
I20260620 23:28:54.398015 5638 sys_catalog.cc:564] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:54.402056 5645 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b78807d2f02c4d739be11ba0cd3af863. Latest consensus state: current_term: 1 leader_uuid: "b78807d2f02c4d739be11ba0cd3af863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78807d2f02c4d739be11ba0cd3af863" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45479 } } }
I20260620 23:28:54.402544 5645 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:54.403106 5644 sys_catalog.cc:455] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b78807d2f02c4d739be11ba0cd3af863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b78807d2f02c4d739be11ba0cd3af863" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45479 } } }
I20260620 23:28:54.403642 5644 sys_catalog.cc:458] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:54.404543 5652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:54.410171 5652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:54.424039 5652 catalog_manager.cc:1347] Generated new cluster ID: a8eb044a00cd4a1aa16cc331926e6356
I20260620 23:28:54.424225 5652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:54.448426 5652 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:54.450157 5652 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:54.466177 5652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b78807d2f02c4d739be11ba0cd3af863: Generated new TSK 0
I20260620 23:28:54.467345 5652 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:28:54.511698 5668 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 23:28:54.511940 5668 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:28:54.513235 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:54.490275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33602 (local address 127.0.96.254:45479)
0620 23:28:54.490525 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:54.490535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:54.491050 (+ 515us) server_negotiation.cc:407] Connection header received
0620 23:28:54.491475 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:54.491479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:54.491566 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:54.491725 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:28:54.492355 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:54.493291 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:54.494899 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:54.495462 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:54.495624 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:54.496270 (+ 646us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:28:54.512407 (+ 16137us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:28:54.512870 (+ 463us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260620 23:28:54.513398 5641 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33602: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 23:28:54.515065 5667 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:28:54.490129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45479 (local address 127.0.0.1:33602)
0620 23:28:54.490903 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:54.490927 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:54.491174 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:54.491759 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:54.491765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:54.491780 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:28:54.492189 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:54.492199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:54.493476 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:54.493481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:54.494734 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:54.494746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:54.495683 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:54.495692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:54.495846 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:54.495932 (+ 86us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:28:54.512775 (+ 16843us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 23:28:54.513232 (+ 457us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.96.254:45479: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":380,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.96.254:45479: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 23:28:54.517810 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5572
I20260620 23:28:54.519456 5633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:54.627183 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5572
2026-06-20T23:28:54Z chronyd exiting
I20260620 23:28:54.681723 387 test_util.cc:175] -----------------------------------------------
I20260620 23:28:54.681866 387 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781998087756853-387-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (890 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 23:28:54.685612 387 mini_oidc.cc:152] Starting JWKS server
I20260620 23:28:54.687580 387 webserver.cc:460] Webserver started at https://127.0.0.1:34491/ using document root <none> and password file <none>
I20260620 23:28:54.687837 387 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:28:54.688601 387 webserver.cc:460] Webserver started at http://127.0.0.1:34937/ using document root <none> and password file <none>
2026-06-20T23:28:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:54Z Disabled control of system clock
I20260620 23:28:54.719137 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38013
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:33089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38013
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/testchainca.cert with env {}
W20260620 23:28:55.059259 5675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:55.059633 5675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:55.059792 5675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:55.067058 5675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:55.067225 5675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:55.067322 5675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:55.067430 5675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:55.067519 5675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:28:55.067621 5675 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:28:55.067704 5675 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34491/jwks.json
I20260620 23:28:55.078195 5675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:33089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:55.079701 5675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:55.081081 5675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:55.094165 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:55.094250 5683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:55.094264 5681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:55.095643 5675 server_base.cc:1034] running on GCE node
I20260620 23:28:55.096534 5675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:55.098153 5675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:55.099426 5675 hybrid_clock.cc:648] HybridClock initialized: now 1781998135099376 us; error 49 us; skew 500 ppm
I20260620 23:28:55.099864 5675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:55.102625 5675 webserver.cc:460] Webserver started at http://127.0.96.254:38729/ using document root <none> and password file <none>
I20260620 23:28:55.103353 5675 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:55.103498 5675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:55.103826 5675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:55.106786 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "f54eb4bc3959405b9b0f5d8576c66719"
format_stamp: "Formatted at 2026-06-20 23:28:55 on dist-test-slave-v34r"
server_key: "8e005152ed1282cd52ce29b92e5b9cc7"
server_key_iv: "cf81fc51e3eb42ed5e28e66dc73f68c9"
server_key_version: "encryptionkey@0"
I20260620 23:28:55.107604 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "f54eb4bc3959405b9b0f5d8576c66719"
format_stamp: "Formatted at 2026-06-20 23:28:55 on dist-test-slave-v34r"
server_key: "8e005152ed1282cd52ce29b92e5b9cc7"
server_key_iv: "cf81fc51e3eb42ed5e28e66dc73f68c9"
server_key_version: "encryptionkey@0"
I20260620 23:28:55.113395 5675 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 23:28:55.117511 5689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:55.119105 5675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:28:55.119316 5675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "f54eb4bc3959405b9b0f5d8576c66719"
format_stamp: "Formatted at 2026-06-20 23:28:55 on dist-test-slave-v34r"
server_key: "8e005152ed1282cd52ce29b92e5b9cc7"
server_key_iv: "cf81fc51e3eb42ed5e28e66dc73f68c9"
server_key_version: "encryptionkey@0"
I20260620 23:28:55.119549 5675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:55.133901 5675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:55.134783 5675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:55.135106 5675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:55.155229 5675 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38013
I20260620 23:28:55.155282 5740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38013 every 8 connection(s)
I20260620 23:28:55.156672 5675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:55.161309 5741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:55.163245 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5675
I20260620 23:28:55.163467 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:55.164163 387 external_mini_cluster.cc:1428] Setting key a42a7b78c738a8e778e403930471b6ed
I20260620 23:28:55.176937 5741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719: Bootstrap starting.
I20260620 23:28:55.180109 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:55.166207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.0.96.254:38013)
0620 23:28:55.167562 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:55.167577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:55.167632 (+ 55us) server_negotiation.cc:407] Connection header received
0620 23:28:55.169527 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:55.169561 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:55.170214 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:55.170657 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:55.171279 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.172710 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.174652 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.175475 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.175640 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:55.177090 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:55.177100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:55.177119 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:55.177872 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:55.177977 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:55.178001 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:55.178109 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:55.178391 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 23:28:55.178866 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":483,"threads_started":1}
I20260620 23:28:55.180675 5741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:55.181924 5741 log.cc:824] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:55.185097 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:28:55.185894 5741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719: No bootstrap required, opened a new log
I20260620 23:28:55.194495 5741 raft_consensus.cc:348] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54eb4bc3959405b9b0f5d8576c66719" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38013 } }
I20260620 23:28:55.195151 5741 raft_consensus.cc:374] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:55.195286 5741 raft_consensus.cc:729] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54eb4bc3959405b9b0f5d8576c66719, State: Initialized, Role: FOLLOWER
I20260620 23:28:55.196103 5741 consensus_queue.cc:260] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54eb4bc3959405b9b0f5d8576c66719" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38013 } }
I20260620 23:28:55.196363 5741 raft_consensus.cc:388] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:55.196485 5741 raft_consensus.cc:482] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:55.196630 5741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:55.199186 5741 raft_consensus.cc:504] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54eb4bc3959405b9b0f5d8576c66719" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38013 } }
I20260620 23:28:55.199841 5741 leader_election.cc:302] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f54eb4bc3959405b9b0f5d8576c66719; no voters:
I20260620 23:28:55.200490 5741 leader_election.cc:288] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:55.200802 5746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:55.201761 5746 raft_consensus.cc:686] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [term 1 LEADER]: Becoming Leader. State: Replica: f54eb4bc3959405b9b0f5d8576c66719, State: Running, Role: LEADER
I20260620 23:28:55.202765 5746 consensus_queue.cc:237] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54eb4bc3959405b9b0f5d8576c66719" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38013 } }
I20260620 23:28:55.203332 5741 sys_catalog.cc:564] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:55.206506 5747 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f54eb4bc3959405b9b0f5d8576c66719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54eb4bc3959405b9b0f5d8576c66719" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38013 } } }
I20260620 23:28:55.207119 5747 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:55.207626 5748 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f54eb4bc3959405b9b0f5d8576c66719. Latest consensus state: current_term: 1 leader_uuid: "f54eb4bc3959405b9b0f5d8576c66719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54eb4bc3959405b9b0f5d8576c66719" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38013 } } }
I20260620 23:28:55.208130 5748 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:55.213382 5755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:55.218165 5755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:55.228138 5755 catalog_manager.cc:1347] Generated new cluster ID: 3044f3212a3645068f29e9ad58ced894
I20260620 23:28:55.228283 5755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:55.243680 5755 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:55.245407 5755 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:55.256242 5755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f54eb4bc3959405b9b0f5d8576c66719: Generated new TSK 0
I20260620 23:28:55.257126 5755 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 23:28:55.307178 5771 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 23:28:55.307356 5771 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:28:55.308638 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:55.280452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.96.254:38013)
0620 23:28:55.280629 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:55.280634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:55.281404 (+ 770us) server_negotiation.cc:407] Connection header received
0620 23:28:55.281679 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:55.281683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:55.281767 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:55.281935 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:28:55.282402 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.283173 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.284499 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.284955 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.285146 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:55.285620 (+ 474us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:28:55.307885 (+ 22265us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:28:55.308301 (+ 416us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46472: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 23:28:55.308779 5744 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46472: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 23:28:55.309298 5770 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:28:55.280522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38013 (local address 127.0.0.1:46472)
0620 23:28:55.281285 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:55.281304 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:55.281481 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:55.281946 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:55.281951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:55.281963 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:28:55.282256 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:55.282265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.283321 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.283329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:55.284351 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:55.284361 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.285165 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.285171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:55.285325 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:55.285357 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:28:55.308458 (+ 23101us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 23:28:55.308827 (+ 369us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.96.254:38013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":384,"threads_started":1}
I20260620 23:28:55.310519 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5675
I20260620 23:28:55.312147 5736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:55.419616 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5675
2026-06-20T23:28:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (792 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 23:28:55.478626 387 mini_oidc.cc:152] Starting JWKS server
I20260620 23:28:55.480610 387 webserver.cc:460] Webserver started at https://127.0.0.1:37351/ using document root <none> and password file <none>
I20260620 23:28:55.480823 387 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 23:28:55.481541 387 webserver.cc:460] Webserver started at http://127.0.0.1:44913/ using document root <none> and password file <none>
2026-06-20T23:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:55Z Disabled control of system clock
I20260620 23:28:55.510535 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34931
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:33523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34931
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37351/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:28:55.834393 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:55.834769 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:55.834916 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:55.842406 5778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:55.842564 5778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:55.842662 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:55.842756 5778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:55.842819 5778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:28:55.842914 5778 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:28:55.842998 5778 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37351/jwks.json
I20260620 23:28:55.852340 5778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:33523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37351/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:55.853796 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:55.855612 5778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:55.868873 5783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:55.868918 5784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:55.868858 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:55.870168 5778 server_base.cc:1034] running on GCE node
I20260620 23:28:55.871042 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:55.872504 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:55.873750 5778 hybrid_clock.cc:648] HybridClock initialized: now 1781998135873691 us; error 59 us; skew 500 ppm
I20260620 23:28:55.874132 5778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:55.876694 5778 webserver.cc:460] Webserver started at http://127.0.96.254:45671/ using document root <none> and password file <none>
I20260620 23:28:55.877477 5778 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:55.877619 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:55.877974 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:55.880856 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "faee618432c540bf826297f624127dab"
format_stamp: "Formatted at 2026-06-20 23:28:55 on dist-test-slave-v34r"
server_key: "2147bb128f61877509efad650e533b4c"
server_key_iv: "718c6a68253d6e8a510711f353016c7c"
server_key_version: "encryptionkey@0"
I20260620 23:28:55.881734 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "faee618432c540bf826297f624127dab"
format_stamp: "Formatted at 2026-06-20 23:28:55 on dist-test-slave-v34r"
server_key: "2147bb128f61877509efad650e533b4c"
server_key_iv: "718c6a68253d6e8a510711f353016c7c"
server_key_version: "encryptionkey@0"
I20260620 23:28:55.887478 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:28:55.891611 5792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:55.893296 5778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:28:55.893498 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "faee618432c540bf826297f624127dab"
format_stamp: "Formatted at 2026-06-20 23:28:55 on dist-test-slave-v34r"
server_key: "2147bb128f61877509efad650e533b4c"
server_key_iv: "718c6a68253d6e8a510711f353016c7c"
server_key_version: "encryptionkey@0"
I20260620 23:28:55.893810 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:55.942189 5778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:55.943080 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:55.943403 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:55.964406 5778 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34931
I20260620 23:28:55.964468 5843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34931 every 8 connection(s)
I20260620 23:28:55.965991 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:55.971730 5844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:55.975553 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5778
I20260620 23:28:55.975805 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:55.976504 387 external_mini_cluster.cc:1428] Setting key 0b6d9138a54bad5f23c5874f24791166
I20260620 23:28:55.987128 5844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab: Bootstrap starting.
I20260620 23:28:55.991554 5844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:55.992105 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:55.978992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.0.96.254:34931)
0620 23:28:55.980294 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:55.980309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:55.980358 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:28:55.982555 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:55.982584 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:55.983174 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:55.983634 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:55.984300 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.985738 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.987093 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:55.987846 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:55.988073 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:55.989517 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:55.989527 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:55.989549 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:55.990171 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:55.990256 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:55.990279 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:55.990425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:55.990698 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:28:55.991051 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":448,"threads_started":1}
I20260620 23:28:55.993137 5844 log.cc:824] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:55.997156 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:28:55.997488 5844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab: No bootstrap required, opened a new log
I20260620 23:28:56.007581 5844 raft_consensus.cc:348] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faee618432c540bf826297f624127dab" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34931 } }
I20260620 23:28:56.008147 5844 raft_consensus.cc:374] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:56.008275 5844 raft_consensus.cc:729] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faee618432c540bf826297f624127dab, State: Initialized, Role: FOLLOWER
I20260620 23:28:56.009150 5844 consensus_queue.cc:260] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faee618432c540bf826297f624127dab" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34931 } }
I20260620 23:28:56.009465 5844 raft_consensus.cc:388] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:56.009627 5844 raft_consensus.cc:482] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:56.009850 5844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:56.013713 5844 raft_consensus.cc:504] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faee618432c540bf826297f624127dab" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34931 } }
I20260620 23:28:56.014537 5844 leader_election.cc:302] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faee618432c540bf826297f624127dab; no voters:
I20260620 23:28:56.015280 5844 leader_election.cc:288] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:56.015456 5849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:56.016479 5849 raft_consensus.cc:686] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [term 1 LEADER]: Becoming Leader. State: Replica: faee618432c540bf826297f624127dab, State: Running, Role: LEADER
I20260620 23:28:56.017177 5849 consensus_queue.cc:237] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faee618432c540bf826297f624127dab" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34931 } }
I20260620 23:28:56.018289 5844 sys_catalog.cc:564] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:56.022233 5851 sys_catalog.cc:455] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [sys.catalog]: SysCatalogTable state changed. Reason: New leader faee618432c540bf826297f624127dab. Latest consensus state: current_term: 1 leader_uuid: "faee618432c540bf826297f624127dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faee618432c540bf826297f624127dab" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34931 } } }
I20260620 23:28:56.023761 5851 sys_catalog.cc:458] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:56.027849 5850 sys_catalog.cc:455] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faee618432c540bf826297f624127dab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faee618432c540bf826297f624127dab" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34931 } } }
I20260620 23:28:56.029021 5850 sys_catalog.cc:458] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:56.031669 5858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:56.037251 5858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:56.046841 5858 catalog_manager.cc:1347] Generated new cluster ID: c3bc5bd320754048ba5a0bf281a1c08e
I20260620 23:28:56.046988 5858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:56.068346 5858 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:56.069860 5858 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:56.080302 5858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faee618432c540bf826297f624127dab: Generated new TSK 0
I20260620 23:28:56.081312 5858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:28:56.114823 5847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:56.103799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.0.96.254:34931)
0620 23:28:56.104094 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:56.104104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:56.104753 (+ 649us) server_negotiation.cc:407] Connection header received
0620 23:28:56.105035 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:56.105046 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:56.105168 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:56.105418 (+ 250us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:28:56.106060 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.107319 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.109235 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.110030 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.110233 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:56.110974 (+ 741us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:28:56.114025 (+ 3051us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:28:56.114461 (+ 436us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":101}
W20260620 23:28:56.114992 5847 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 23:28:56.116326 5873 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:28:56.103702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34931 (local address 127.0.0.1:52716)
0620 23:28:56.104617 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:56.104634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:56.104771 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:56.105509 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:56.105513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:56.105525 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:28:56.105829 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:56.105839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.107608 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.107613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:56.108886 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:56.108900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.110282 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.110287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:56.110531 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:56.110596 (+ 65us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:28:56.114303 (+ 3707us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 23:28:56.115902 (+ 1599us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.96.254:34931: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":1249,"thread_start_us":605,"threads_started":1}
I20260620 23:28:56.118971 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5778
I20260620 23:28:56.120705 5839 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 23:28:56.127722 5874 webserver.cc:533] Webserver: sslize failed:
E20260620 23:28:56.127838 5874 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 23:28:56.217664 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5778
2026-06-20T23:28:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (796 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T23:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:28:56Z Disabled control of system clock
I20260620 23:28:56.311235 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:37921
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:37921
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/testchainca.cert with env {}
W20260620 23:28:56.716181 5879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:28:56.716588 5879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:28:56.716687 5879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:28:56.724838 5879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:28:56.725069 5879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:28:56.725147 5879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:28:56.725214 5879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:28:56.725270 5879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 23:28:56.725343 5879 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 23:28:56.725386 5879 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 23:28:56.737056 5879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:37921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:28:56.738767 5879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:28:56.740427 5879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:28:56.755028 5884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:56.755115 5885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:28:56.755127 5887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:28:56.756539 5879 server_base.cc:1034] running on GCE node
I20260620 23:28:56.757829 5879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:28:56.759852 5879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:28:56.761145 5879 hybrid_clock.cc:648] HybridClock initialized: now 1781998136761088 us; error 72 us; skew 500 ppm
I20260620 23:28:56.761639 5879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:28:56.764636 5879 webserver.cc:460] Webserver started at http://127.0.96.254:34345/ using document root <none> and password file <none>
I20260620 23:28:56.765662 5879 fs_manager.cc:359] Metadata directory not provided
I20260620 23:28:56.765830 5879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:28:56.766239 5879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:28:56.769980 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "36c11a31aaab42428c30a987c14d8e92"
format_stamp: "Formatted at 2026-06-20 23:28:56 on dist-test-slave-v34r"
server_key: "ea5f49e4709eda99b697bf8cb17b639e"
server_key_iv: "99b45242542db96632b9bba06d329be9"
server_key_version: "encryptionkey@0"
I20260620 23:28:56.770972 5879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "36c11a31aaab42428c30a987c14d8e92"
format_stamp: "Formatted at 2026-06-20 23:28:56 on dist-test-slave-v34r"
server_key: "ea5f49e4709eda99b697bf8cb17b639e"
server_key_iv: "99b45242542db96632b9bba06d329be9"
server_key_version: "encryptionkey@0"
I20260620 23:28:56.778048 5879 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:28:56.782707 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:28:56.784602 5879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:28:56.784893 5879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "36c11a31aaab42428c30a987c14d8e92"
format_stamp: "Formatted at 2026-06-20 23:28:56 on dist-test-slave-v34r"
server_key: "ea5f49e4709eda99b697bf8cb17b639e"
server_key_iv: "99b45242542db96632b9bba06d329be9"
server_key_version: "encryptionkey@0"
I20260620 23:28:56.785236 5879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:28:56.803028 5879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:28:56.804251 5879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:28:56.804625 5879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:28:56.826889 5879 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:37921
I20260620 23:28:56.826910 5944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:37921 every 8 connection(s)
I20260620 23:28:56.828624 5879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:28:56.832793 5945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:28:56.838656 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 5879
I20260620 23:28:56.838919 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:28:56.839648 387 external_mini_cluster.cc:1428] Setting key c07563ce5ab4f0b39cbd95a69b5149b4
I20260620 23:28:56.844947 5945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92: Bootstrap starting.
I20260620 23:28:56.848332 5945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92: Neither blocks nor log segments found. Creating new log.
I20260620 23:28:56.849520 5945 log.cc:824] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92: Log is configured to *not* fsync() on all Append() calls
I20260620 23:28:56.853199 5945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92: No bootstrap required, opened a new log
I20260620 23:28:56.857220 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:56.842073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.0.96.254:37921)
0620 23:28:56.843547 (+ 1474us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:56.843567 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:56.843632 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:28:56.845975 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:56.846007 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:56.846688 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:56.847212 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:28:56.847917 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.849543 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.851307 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.852151 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.852363 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:56.853961 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:28:56.853972 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:28:56.853994 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:28:56.854706 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:28:56.854819 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:28:56.854847 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:28:56.855050 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 23:28:56.855333 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:28:56.855789 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":488,"threads_started":1}
I20260620 23:28:56.862279 5945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c11a31aaab42428c30a987c14d8e92" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:28:56.862766 5945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:28:56.862766 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:28:56.862872 5945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36c11a31aaab42428c30a987c14d8e92, State: Initialized, Role: FOLLOWER
I20260620 23:28:56.863518 5945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c11a31aaab42428c30a987c14d8e92" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:28:56.863772 5945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:28:56.863925 5945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:28:56.864114 5945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:28:56.866638 5945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c11a31aaab42428c30a987c14d8e92" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:28:56.867244 5945 leader_election.cc:302] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36c11a31aaab42428c30a987c14d8e92; no voters:
I20260620 23:28:56.867884 5945 leader_election.cc:288] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:28:56.868148 5950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:28:56.869117 5950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [term 1 LEADER]: Becoming Leader. State: Replica: 36c11a31aaab42428c30a987c14d8e92, State: Running, Role: LEADER
I20260620 23:28:56.869853 5950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c11a31aaab42428c30a987c14d8e92" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:28:56.870138 5945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:28:56.873186 5952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36c11a31aaab42428c30a987c14d8e92. Latest consensus state: current_term: 1 leader_uuid: "36c11a31aaab42428c30a987c14d8e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c11a31aaab42428c30a987c14d8e92" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } } }
I20260620 23:28:56.873930 5952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:56.873821 5951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36c11a31aaab42428c30a987c14d8e92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36c11a31aaab42428c30a987c14d8e92" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } } }
I20260620 23:28:56.874171 5951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92 [sys.catalog]: This master's current role is: LEADER
I20260620 23:28:56.875463 5959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:28:56.882018 5959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:28:56.893829 5959 catalog_manager.cc:1347] Generated new cluster ID: 6b47edb8b28d4517b8dbaaa4f4df557d
I20260620 23:28:56.894009 5959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:28:56.915355 5959 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:28:56.917289 5959 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:28:56.930297 5959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36c11a31aaab42428c30a987c14d8e92: Generated new TSK 0
I20260620 23:28:56.931535 5959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:28:57.821280 5948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:28:56.954639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.0.96.254:37921)
0620 23:28:56.954958 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 23:28:56.954967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:28:56.955550 (+ 583us) server_negotiation.cc:407] Connection header received
0620 23:28:56.955846 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:28:56.955850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:28:56.955973 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:28:56.956162 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0620 23:28:56.956761 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.957699 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.959836 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.960364 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.960485 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:56.961207 (+ 722us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 23:28:57.820453 (+859246us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:28:57.820882 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39914: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":100}
W20260620 23:28:57.821408 5948 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39914: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 23:28:57.821857 5974 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:28:56.954578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37921 (local address 127.0.0.1:39914)
0620 23:28:56.955382 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 23:28:56.955406 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:28:56.955622 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:28:56.956214 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:28:56.956219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:28:56.956234 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0620 23:28:56.956573 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:56.956586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.957872 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.957878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:56.959653 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:28:56.959668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:28:56.960570 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:28:56.960577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:28:56.960786 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:28:56.960835 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 23:28:57.820832 (+859997us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 23:28:57.821194 (+ 362us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.96.254:37921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":422,"threads_started":1}
I20260620 23:28:57.823066 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 5879
I20260620 23:28:57.824491 5940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:28:57.940722 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 5879
2026-06-20T23:28:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1704 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (482 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1599 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:00 dist-test-slave-v34r krb5kdc[5990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:00 dist-test-slave-v34r krb5kdc[5990](info): set up 2 sockets
Jun 20 23:29:00 dist-test-slave-v34r krb5kdc[5990](info): commencing operation
krb5kdc: starting...
W20260620 23:29:01.126008 387 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:01 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:29:01.367048 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:46697
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:01.718407 6003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:01.718735 6003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:01.718827 6003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:01.726256 6003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:01.726379 6003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:01.726430 6003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:01.726496 6003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:01.726548 6003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:01.736680 6003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:46697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:01.738067 6003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:01.739293 6003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:01.752882 6008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:01.752766 6009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:01.752794 6011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:01.753875 6003 server_base.cc:1034] running on GCE node
I20260620 23:29:01.754897 6003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:01.756568 6003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:01.757870 6003 hybrid_clock.cc:648] HybridClock initialized: now 1781998141757785 us; error 92 us; skew 500 ppm
I20260620 23:29:01.758293 6003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:01 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:01.762013 6003 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:29:01.763844 6003 webserver.cc:460] Webserver started at http://127.0.96.254:43061/ using document root <none> and password file <none>
I20260620 23:29:01.764632 6003 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:01.764789 6003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:01.765199 6003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:01.768301 6003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "949b083cb62b423a91237b51eb6a0bb3"
format_stamp: "Formatted at 2026-06-20 23:29:01 on dist-test-slave-v34r"
server_key: "18f896ff243b4c7bbb7bc4af5319a538"
server_key_iv: "bd166b737250701e0d612603308ed8a7"
server_key_version: "encryptionkey@0"
I20260620 23:29:01.769193 6003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "949b083cb62b423a91237b51eb6a0bb3"
format_stamp: "Formatted at 2026-06-20 23:29:01 on dist-test-slave-v34r"
server_key: "18f896ff243b4c7bbb7bc4af5319a538"
server_key_iv: "bd166b737250701e0d612603308ed8a7"
server_key_version: "encryptionkey@0"
I20260620 23:29:01.774909 6003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:29:01.779120 6018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:01.780784 6003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 23:29:01.781100 6003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "949b083cb62b423a91237b51eb6a0bb3"
format_stamp: "Formatted at 2026-06-20 23:29:01 on dist-test-slave-v34r"
server_key: "18f896ff243b4c7bbb7bc4af5319a538"
server_key_iv: "bd166b737250701e0d612603308ed8a7"
server_key_version: "encryptionkey@0"
I20260620 23:29:01.781306 6003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:01.799459 6003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:01.803849 6003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:01.804140 6003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:01.825436 6003 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:46697
I20260620 23:29:01.825533 6069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:46697 every 8 connection(s)
I20260620 23:29:01.826911 6003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:01.831758 6070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:01.833581 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6003
I20260620 23:29:01.833791 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:01.834456 387 external_mini_cluster.cc:1428] Setting key 32d2bcd50e1166519151ee8579338f12
I20260620 23:29:01.850173 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3: Bootstrap starting.
Jun 20 23:29:01 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:01.855197 6070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:01.857044 6070 log.cc:824] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:01.861635 6070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3: No bootstrap required, opened a new log
I20260620 23:29:01.863091 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:01.836785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.0.96.254:46697)
0620 23:29:01.837939 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:01.837960 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:01.838048 (+ 88us) server_negotiation.cc:407] Connection header received
0620 23:29:01.840149 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:01.840189 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:01.840958 (+ 769us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:01.841613 (+ 655us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:01.842922 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:01.845136 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:01.846593 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:01.847977 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:01.848233 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:01.852573 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:01.852588 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:01.852616 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:01.852694 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:01.857482 (+ 4788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:01.858553 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:01.858562 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:01.858571 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:01.858874 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:01.859346 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:01.859354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:01.859357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:01.860173 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:01.860511 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:01.860961 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 23:29:01.861539 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":529,"threads_started":1}
I20260620 23:29:01.871519 6070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949b083cb62b423a91237b51eb6a0bb3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46697 } }
I20260620 23:29:01.871984 6070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:01.872076 6070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949b083cb62b423a91237b51eb6a0bb3, State: Initialized, Role: FOLLOWER
I20260620 23:29:01.872722 6070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949b083cb62b423a91237b51eb6a0bb3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46697 } }
I20260620 23:29:01.872947 6070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:01.873131 6070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:01.873303 6070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:01.876219 6070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949b083cb62b423a91237b51eb6a0bb3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46697 } }
I20260620 23:29:01.876863 6070 leader_election.cc:302] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 949b083cb62b423a91237b51eb6a0bb3; no voters:
I20260620 23:29:01.877692 6070 leader_election.cc:288] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:01.877920 6075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:01.880296 6075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [term 1 LEADER]: Becoming Leader. State: Replica: 949b083cb62b423a91237b51eb6a0bb3, State: Running, Role: LEADER
I20260620 23:29:01.880961 6075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949b083cb62b423a91237b51eb6a0bb3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46697 } }
I20260620 23:29:01.881765 6070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:01.884543 6076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "949b083cb62b423a91237b51eb6a0bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949b083cb62b423a91237b51eb6a0bb3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46697 } } }
I20260620 23:29:01.885077 6076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:01.885916 6077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949b083cb62b423a91237b51eb6a0bb3. Latest consensus state: current_term: 1 leader_uuid: "949b083cb62b423a91237b51eb6a0bb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949b083cb62b423a91237b51eb6a0bb3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46697 } } }
I20260620 23:29:01.886176 6083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:01.886358 6077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:01.893599 6083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:01.904794 6083 catalog_manager.cc:1347] Generated new cluster ID: fa0a0b3ed8d444019ea32e09eeea78e8
I20260620 23:29:01.904942 6083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:01.921291 6083 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:01.923022 6083 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:01.931200 6083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 949b083cb62b423a91237b51eb6a0bb3: Generated new TSK 0
I20260620 23:29:01.932490 6083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:29:02.003165 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46697
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:02.443670 6096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:02.444041 6096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:02.444219 6096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:02.452754 6096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:02.452922 6096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:02.453172 6096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:02.464843 6096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:02.466755 6096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:02.468446 6096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:02.485064 6101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:02.485116 6102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:02.486049 6104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:02.486562 6096 server_base.cc:1034] running on GCE node
I20260620 23:29:02.487509 6096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:02.489307 6096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:02.490605 6096 hybrid_clock.cc:648] HybridClock initialized: now 1781998142490531 us; error 83 us; skew 500 ppm
I20260620 23:29:02.491078 6096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:02 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:02.495431 6096 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:29:02.497774 6096 webserver.cc:460] Webserver started at http://127.0.96.193:43081/ using document root <none> and password file <none>
I20260620 23:29:02.498786 6096 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:02.498977 6096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:02.499373 6096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:02.503298 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "b1a3ff22cac548329cded3da29bebd79"
format_stamp: "Formatted at 2026-06-20 23:29:02 on dist-test-slave-v34r"
server_key: "0d80873906b788e1dbd25a8434c581df"
server_key_iv: "2eaf993680c363bbedd18b2b0ebdfa0f"
server_key_version: "encryptionkey@0"
I20260620 23:29:02.504317 6096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "b1a3ff22cac548329cded3da29bebd79"
format_stamp: "Formatted at 2026-06-20 23:29:02 on dist-test-slave-v34r"
server_key: "0d80873906b788e1dbd25a8434c581df"
server_key_iv: "2eaf993680c363bbedd18b2b0ebdfa0f"
server_key_version: "encryptionkey@0"
I20260620 23:29:02.511471 6096 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 23:29:02.516340 6111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:02.518283 6096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:02.518535 6096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "b1a3ff22cac548329cded3da29bebd79"
format_stamp: "Formatted at 2026-06-20 23:29:02 on dist-test-slave-v34r"
server_key: "0d80873906b788e1dbd25a8434c581df"
server_key_iv: "2eaf993680c363bbedd18b2b0ebdfa0f"
server_key_version: "encryptionkey@0"
I20260620 23:29:02.518807 6096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:02.566416 6096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:02.571691 6096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:02.572078 6096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:02.573380 6096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:02.575130 6096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:02.575248 6096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:02.575423 6096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:02.575532 6096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:02.616000 6096 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:38077
I20260620 23:29:02.616117 6223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:38077 every 8 connection(s)
I20260620 23:29:02.617733 6096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:02.619649 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6096
I20260620 23:29:02.619912 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:02.620676 387 external_mini_cluster.cc:1428] Setting key 27aaad132c9da2cbf1f870ae1eefabf5
Jun 20 23:29:02 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:02.639834 6226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:02.621003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:34977 (local address 127.0.96.254:46697)
0620 23:29:02.621843 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:02.621849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:02.623218 (+ 1369us) server_negotiation.cc:407] Connection header received
0620 23:29:02.625105 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:02.625110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:02.625211 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:02.625336 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:02.626998 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:02.627756 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:02.629771 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:02.630179 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:02.630322 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:02.633954 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:02.633960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:02.633963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:02.633995 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:02.636652 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:02.637448 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:02.637458 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:02.637461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:02.637573 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:02.638005 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:02.638010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:02.638013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:02.638291 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:02.638445 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:02.639204 (+ 759us) server_negotiation.cc:299] Negotiation successful
0620 23:29:02.639457 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":492,"threads_started":1}
I20260620 23:29:02.640937 6227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:02.621763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:46697 (local address 127.0.96.193:34977)
0620 23:29:02.622707 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:02.622758 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:02.624773 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:02.625590 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:02.625600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:02.626204 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:02.626821 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:02.626837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:02.627938 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:02.627942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:02.629601 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:02.629614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:02.630419 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:02.630427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:02.630686 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:02.631453 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:02.631476 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:02.633707 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:02.636807 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:02.636816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:02.636834 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:02.637319 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:02.637685 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:02.637690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:02.637694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:02.637885 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:02.638415 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:02.638423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:02.638817 (+ 394us) client_negotiation.cc:769] Sending connection context
0620 23:29:02.639091 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 23:29:02.639638 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":404,"threads_started":1}
I20260620 23:29:02.642866 6224 heartbeater.cc:347] Connected to a master server at 127.0.96.254:46697
I20260620 23:29:02.643185 6224 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:02.643957 6224 heartbeater.cc:511] Master 127.0.96.254:46697 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:29:02.646317 6035 ts_manager.cc:194] Registered new tserver with Master: b1a3ff22cac548329cded3da29bebd79 (127.0.96.193:38077)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:29:02.648134 6035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:34977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:29:02.674629 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46697
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:03.094974 6230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:03.095441 6230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:03.095649 6230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:03.104950 6230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:03.105342 6230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:03.105568 6230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:03.117211 6230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:03.119110 6230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:03.120886 6230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:03.139060 6235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:03.139174 6236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:03.139238 6238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:03.140151 6230 server_base.cc:1034] running on GCE node
I20260620 23:29:03.141050 6230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:03.142336 6230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:03.143568 6230 hybrid_clock.cc:648] HybridClock initialized: now 1781998143143525 us; error 59 us; skew 500 ppm
I20260620 23:29:03.144003 6230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.147961 6230 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:29:03.150002 6230 webserver.cc:460] Webserver started at http://127.0.96.194:34015/ using document root <none> and password file <none>
I20260620 23:29:03.150943 6230 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:03.151119 6230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:03.151497 6230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:03.155682 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "a16465d53bfc444ab3ab762c0fd2f3e5"
format_stamp: "Formatted at 2026-06-20 23:29:03 on dist-test-slave-v34r"
server_key: "257ffeaac227480c1e549785ce3a4f79"
server_key_iv: "84f80d79422cb71dd8a6e69b62372f3d"
server_key_version: "encryptionkey@0"
I20260620 23:29:03.156805 6230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "a16465d53bfc444ab3ab762c0fd2f3e5"
format_stamp: "Formatted at 2026-06-20 23:29:03 on dist-test-slave-v34r"
server_key: "257ffeaac227480c1e549785ce3a4f79"
server_key_iv: "84f80d79422cb71dd8a6e69b62372f3d"
server_key_version: "encryptionkey@0"
I20260620 23:29:03.163723 6230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:29:03.168386 6245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:03.170445 6230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:03.170717 6230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "a16465d53bfc444ab3ab762c0fd2f3e5"
format_stamp: "Formatted at 2026-06-20 23:29:03 on dist-test-slave-v34r"
server_key: "257ffeaac227480c1e549785ce3a4f79"
server_key_iv: "84f80d79422cb71dd8a6e69b62372f3d"
server_key_version: "encryptionkey@0"
I20260620 23:29:03.171020 6230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:03.198690 6230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:03.204463 6230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:03.204842 6230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:03.206213 6230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:03.207969 6230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:03.208089 6230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:03.208259 6230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:03.208343 6230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:03.248438 6230 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:38027
I20260620 23:29:03.248493 6357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:38027 every 8 connection(s)
I20260620 23:29:03.250222 6230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:03.250640 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6230
I20260620 23:29:03.250875 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:03.251626 387 external_mini_cluster.cc:1428] Setting key 0f55d480e80d6226347ebdafe4106553
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:29:03.276654 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.253578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:51845 (local address 127.0.96.254:46697)
0620 23:29:03.254110 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.254115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.255295 (+ 1180us) server_negotiation.cc:407] Connection header received
0620 23:29:03.257297 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.257307 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.257430 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.257610 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.259157 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.260581 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.262704 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.263422 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.263565 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.269274 (+ 5709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:03.269280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:03.269284 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:03.269334 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:03.272537 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.273748 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.273753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.273756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.273914 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.274438 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.274441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.274443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.274756 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:03.274872 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:03.275950 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0620 23:29:03.276165 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":303,"threads_started":1}
I20260620 23:29:03.277845 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.254195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:46697 (local address 127.0.96.194:51845)
0620 23:29:03.255106 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:03.255157 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:03.256936 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:03.257823 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:03.257831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:03.258349 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:03.258951 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:03.258967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.260803 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.260808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:03.262478 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:03.262493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.263733 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.263743 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:03.264098 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.265661 (+ 1563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:03.265699 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:03.268891 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:03.272753 (+ 3862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:03.272767 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:03.272792 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:03.273574 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:03.274044 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:03.274049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:03.274060 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:03.274314 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:03.274898 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:03.274910 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:03.275471 (+ 561us) client_negotiation.cc:769] Sending connection context
0620 23:29:03.275852 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 23:29:03.276470 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":389,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:29:03.279719 6358 heartbeater.cc:347] Connected to a master server at 127.0.96.254:46697
I20260620 23:29:03.280109 6358 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:03.280890 6358 heartbeater.cc:511] Master 127.0.96.254:46697 requested a full tablet report, sending...
I20260620 23:29:03.282441 6035 ts_manager.cc:194] Registered new tserver with Master: a16465d53bfc444ab3ab762c0fd2f3e5 (127.0.96.194:38027)
I20260620 23:29:03.283274 6035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:51845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:29:03.306484 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46697
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:03.652207 6224 heartbeater.cc:503] Master 127.0.96.254:46697 was elected leader, sending a full tablet report...
W20260620 23:29:03.656967 6364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:03.657326 6364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:03.657478 6364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:03.664798 6364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:03.664913 6364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:03.665122 6364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:03.674994 6364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:03.676441 6364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:03.677832 6364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:03.692946 6372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:03.692916 6370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:03.693049 6369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:03.694105 6364 server_base.cc:1034] running on GCE node
I20260620 23:29:03.694801 6364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:03.695824 6364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:03.697077 6364 hybrid_clock.cc:648] HybridClock initialized: now 1781998143697014 us; error 49 us; skew 500 ppm
I20260620 23:29:03.697422 6364 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.700695 6364 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:29:03.702445 6364 webserver.cc:460] Webserver started at http://127.0.96.195:41425/ using document root <none> and password file <none>
I20260620 23:29:03.703146 6364 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:03.703277 6364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:03.703590 6364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:03.706565 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "0065cd4910d14f3aabf67a98a51e1f52"
format_stamp: "Formatted at 2026-06-20 23:29:03 on dist-test-slave-v34r"
server_key: "d2646e4c7ea5084ae53aa4b3549be433"
server_key_iv: "78ef2337221bc9dcb5e17f35291333c7"
server_key_version: "encryptionkey@0"
I20260620 23:29:03.707365 6364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "0065cd4910d14f3aabf67a98a51e1f52"
format_stamp: "Formatted at 2026-06-20 23:29:03 on dist-test-slave-v34r"
server_key: "d2646e4c7ea5084ae53aa4b3549be433"
server_key_iv: "78ef2337221bc9dcb5e17f35291333c7"
server_key_version: "encryptionkey@0"
I20260620 23:29:03.713156 6364 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 23:29:03.717110 6379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:03.718694 6364 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:29:03.718899 6364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "0065cd4910d14f3aabf67a98a51e1f52"
format_stamp: "Formatted at 2026-06-20 23:29:03 on dist-test-slave-v34r"
server_key: "d2646e4c7ea5084ae53aa4b3549be433"
server_key_iv: "78ef2337221bc9dcb5e17f35291333c7"
server_key_version: "encryptionkey@0"
I20260620 23:29:03.719134 6364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:03.732779 6364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:03.737166 6364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:03.737458 6364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:03.738435 6364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:03.739890 6364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:03.739995 6364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:03.740149 6364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:03.740227 6364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:03.778887 6364 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:38445
I20260620 23:29:03.778906 6491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:38445 every 8 connection(s)
I20260620 23:29:03.780241 6364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:03.782817 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6364
I20260620 23:29:03.783028 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:03.783591 387 external_mini_cluster.cc:1428] Setting key f84e4466548f2260cf108e997eb1ce19
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:03.805243 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.783842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:49399 (local address 127.0.96.254:46697)
0620 23:29:03.784453 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.784459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.786040 (+ 1581us) server_negotiation.cc:407] Connection header received
0620 23:29:03.788523 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.788529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.788659 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.788791 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.790766 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.791619 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.793475 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.793927 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.794076 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.798341 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:03.798348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:03.798351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:03.798400 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:03.801596 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.802509 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.802515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.802518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.802650 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.803145 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.803150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.803152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.803476 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:03.803658 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:03.804499 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 23:29:03.804761 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":278,"threads_started":1}
I20260620 23:29:03.806483 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.784436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:46697 (local address 127.0.96.195:49399)
0620 23:29:03.785817 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:03.785900 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:03.788157 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:03.789164 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:03.789175 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:03.789844 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:03.790556 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:03.790576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.791808 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.791818 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:03.793315 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:03.793329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.794185 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.794192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:03.794467 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.795263 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:03.795291 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:03.798012 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:03.801793 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:03.801804 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:03.801833 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:03.802358 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:03.802771 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:03.802776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:03.802780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:03.803009 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:03.803616 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:03.803626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:03.804086 (+ 460us) client_negotiation.cc:769] Sending connection context
0620 23:29:03.804392 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 23:29:03.805090 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":546,"threads_started":1}
I20260620 23:29:03.808068 6492 heartbeater.cc:347] Connected to a master server at 127.0.96.254:46697
I20260620 23:29:03.808490 6492 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:03.809664 6492 heartbeater.cc:511] Master 127.0.96.254:46697 requested a full tablet report, sending...
I20260620 23:29:03.811576 6035 ts_manager.cc:194] Registered new tserver with Master: 0065cd4910d14f3aabf67a98a51e1f52 (127.0.96.195:38445)
I20260620 23:29:03.812849 6035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:49399
I20260620 23:29:03.813227 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/test-admin.keytab.
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.840149 387 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 23:29:03.846077 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:03.843350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.0.96.193:38077)
0620 23:29:03.844046 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.844054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.844087 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:29:03.844157 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.844163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.844373 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.844523 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.845540 (+ 1017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58158: BlockingRecv error: recv got EOF from 127.0.0.1:58158 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":362,"threads_started":1}
W20260620 23:29:03.846490 387 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.848004 387 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.849421 387 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:29:03.865414 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.851214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.96.193:38077)
0620 23:29:03.851468 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.851475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.851797 (+ 322us) server_negotiation.cc:407] Connection header received
0620 23:29:03.852021 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.852025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.852085 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.852186 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.853204 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.854170 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.855375 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.855962 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.856133 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.859701 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:03.859712 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:03.859728 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:03.859761 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:03.862694 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.863409 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.863416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.863418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.863531 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.863893 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.863900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.863903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.864243 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:03.864534 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:03.864849 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 23:29:03.865143 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260620 23:29:03.872267 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:03.870210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.0.96.193:38077)
0620 23:29:03.870515 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.870523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.870807 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:29:03.870966 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.870970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.871048 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.871147 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.871943 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58190: BlockingRecv error: recv got EOF from 127.0.0.1:58190 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 23:29:03.872304 387 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.873929 387 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.875306 387 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:29:03.891896 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.877086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.0.96.193:38077)
0620 23:29:03.877346 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.877352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.877647 (+ 295us) server_negotiation.cc:407] Connection header received
0620 23:29:03.877855 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.877861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.877939 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.878048 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.878956 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.879725 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.880941 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.881486 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.881693 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.885872 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:03.885879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:03.885884 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:03.886006 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:03.889199 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.889952 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.889956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.889959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.890127 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.890524 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.890528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.890530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.890894 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:03.891129 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:03.891391 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 23:29:03.891594 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260620 23:29:03.897404 6500 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:03.895189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58202 (local address 127.0.96.193:38077)
0620 23:29:03.895465 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.895472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.895703 (+ 231us) server_negotiation.cc:407] Connection header received
0620 23:29:03.895890 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.895894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.895985 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.896086 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.897099 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58202: BlockingRecv error: recv got EOF from 127.0.0.1:58202 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260620 23:29:03.897526 387 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781998087756853-387-0/krb5kdc/krb5cc' of type FILE
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.899310 387 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:03.900833 387 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 23:29:03 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:29:03.917332 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.902596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.96.193:38077)
0620 23:29:03.902904 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.902912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.903215 (+ 303us) server_negotiation.cc:407] Connection header received
0620 23:29:03.903451 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.903456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.903528 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.903646 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.904690 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.905567 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.906954 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.907408 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.907551 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.911513 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:03.911518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:03.911522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:03.911560 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:03.914473 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.915183 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.915190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.915194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.915342 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.915747 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.915754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.915757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.916076 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:03.916442 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:03.916705 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 23:29:03.916935 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:29:03.919346 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6096
I20260620 23:29:03.933398 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:03.920272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.0.96.193:38077)
0620 23:29:03.920486 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:03.920490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:03.920857 (+ 367us) server_negotiation.cc:407] Connection header received
0620 23:29:03.921089 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:03.921093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:03.921155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:03.921249 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:03.922379 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.923141 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.924612 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:03.925171 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:03.925289 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:03.927595 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:03.927600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:03.927604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:03.927636 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:03.930290 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.931069 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.931074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.931077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.931219 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:03.931666 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:03.931669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:03.931671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:03.931929 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:03.932116 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:03.932650 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 23:29:03.932960 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 23:29:03.934134 6219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:04.021960 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6096
I20260620 23:29:04.040786 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6230
Jun 20 23:29:04 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:29:04.058391 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:04.042069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.0.96.194:38027)
0620 23:29:04.042768 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:04.042776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:04.042802 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:29:04.042861 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:04.042867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:04.043086 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:04.043208 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:04.044480 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:04.045590 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:04.047040 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:04.047570 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:04.047677 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:04.052218 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:04.052230 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:04.052245 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:04.052290 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:04.055243 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:04.056159 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:04.056166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:04.056168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:04.056311 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:04.056754 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:04.056757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:04.056759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:04.057254 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:04.057483 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:04.057875 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 23:29:04.058103 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":360,"threads_started":1}
I20260620 23:29:04.060056 6353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:04.152088 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6230
I20260620 23:29:04.168500 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6364
Jun 20 23:29:04 dist-test-slave-v34r krb5kdc[5990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:29:04.185690 6515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:04.169750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.0.96.195:38445)
0620 23:29:04.170395 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:04.170403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:04.170432 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:29:04.170496 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:04.170502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:04.170739 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:04.170910 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:04.172006 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:04.173148 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:04.174373 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:04.174905 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:04.175046 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:04.179185 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:04.179196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:04.179210 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:04.179243 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:04.182616 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:04.183364 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:04.183377 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:04.183380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:04.183493 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:04.183886 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:04.183898 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:04.183900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:04.184310 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:04.184649 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:04.184914 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 23:29:04.185367 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":315,"threads_started":1}
I20260620 23:29:04.187515 6487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:04.279438 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6364
I20260620 23:29:04.295836 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6003
I20260620 23:29:04.297380 6065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:04.401464 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6003
2026-06-20T23:29:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4381 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:04 dist-test-slave-v34r krb5kdc[6520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:04 dist-test-slave-v34r krb5kdc[6520](info): set up 2 sockets
Jun 20 23:29:04 dist-test-slave-v34r krb5kdc[6520](info): commencing operation
krb5kdc: starting...
W20260620 23:29:05.505795 387 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:05 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab.
Entry for principal oryx/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab.
I20260620 23:29:05.746292 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:45755
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab
--principal=oryx/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:06.149784 6533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:06.150266 6533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:06.150465 6533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:06.159338 6533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:06.159555 6533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:06.159677 6533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:06.159790 6533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:06.159876 6533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:06.172801 6533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab
--principal=oryx/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:06.174731 6533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:06.176584 6533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:06.192093 6538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:06.192059 6539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:06.192230 6541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:06.193094 6533 server_base.cc:1034] running on GCE node
I20260620 23:29:06.194147 6533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:06.196226 6533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:06.197542 6533 hybrid_clock.cc:648] HybridClock initialized: now 1781998146197469 us; error 59 us; skew 500 ppm
I20260620 23:29:06.198029 6533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:06 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:06.205471 6533 init.cc:377] Logged in from keytab as oryx/127.0.96.254@KRBTEST.COM (short username oryx)
I20260620 23:29:06.207464 6533 webserver.cc:460] Webserver started at http://127.0.96.254:43407/ using document root <none> and password file <none>
I20260620 23:29:06.208410 6533 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:06.208595 6533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:06.209048 6533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:06.212857 6533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "36b55db1ddc34f209ac28fe8639a9a08"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "ed1c33e6788a1e841378042f65b421b6"
server_key_iv: "f649cac84a7618c928074770e156137b"
server_key_version: "encryptionkey@0"
I20260620 23:29:06.213922 6533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "36b55db1ddc34f209ac28fe8639a9a08"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "ed1c33e6788a1e841378042f65b421b6"
server_key_iv: "f649cac84a7618c928074770e156137b"
server_key_version: "encryptionkey@0"
I20260620 23:29:06.220742 6533 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:29:06.225142 6548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:06.226991 6533 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:06.227267 6533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "36b55db1ddc34f209ac28fe8639a9a08"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "ed1c33e6788a1e841378042f65b421b6"
server_key_iv: "f649cac84a7618c928074770e156137b"
server_key_version: "encryptionkey@0"
I20260620 23:29:06.227550 6533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:06.262324 6533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:06.268021 6533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:06.268635 6533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:06.291200 6533 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:45755
I20260620 23:29:06.291225 6599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:45755 every 8 connection(s)
I20260620 23:29:06.293036 6533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:06.297760 6600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:06.302541 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6533
I20260620 23:29:06.302789 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:06.303704 387 external_mini_cluster.cc:1428] Setting key c73619cc52a034ae39522e054f9e0b9c
I20260620 23:29:06.311111 6600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Bootstrap starting.
I20260620 23:29:06.315017 6600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:06.316311 6600 log.cc:824] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:06.320605 6600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: No bootstrap required, opened a new log
Jun 20 23:29:06 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:06.329171 6600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } }
I20260620 23:29:06.329551 6600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:06.329696 6600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b55db1ddc34f209ac28fe8639a9a08, State: Initialized, Role: FOLLOWER
I20260620 23:29:06.330421 6600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } }
I20260620 23:29:06.330686 6600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:06.330863 6600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:06.331056 6600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:06.333093 6603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:06.306508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.0.96.254:45755)
0620 23:29:06.307714 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:06.307731 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:06.307789 (+ 58us) server_negotiation.cc:407] Connection header received
0620 23:29:06.309994 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:06.310027 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:06.310598 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:06.311068 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:06.312899 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:06.314507 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:06.316554 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:06.317889 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:06.318171 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:06.323285 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:06.323299 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:06.323321 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:06.323375 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:06.327335 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:06.328271 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:06.328284 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:06.328292 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:06.328504 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:06.328925 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:06.328929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:06.328931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:06.329825 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:06.330186 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:06.330696 (+ 510us) server_negotiation.cc:299] Negotiation successful
0620 23:29:06.331467 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":631,"threads_started":1}
I20260620 23:29:06.333931 6600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } }
I20260620 23:29:06.334494 6600 leader_election.cc:302] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36b55db1ddc34f209ac28fe8639a9a08; no voters:
I20260620 23:29:06.335264 6600 leader_election.cc:288] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:06.335374 6605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:06.338294 6600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:06.338811 6605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 LEADER]: Becoming Leader. State: Replica: 36b55db1ddc34f209ac28fe8639a9a08, State: Running, Role: LEADER
I20260620 23:29:06.339442 6605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } }
I20260620 23:29:06.343209 6606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } }
I20260620 23:29:06.343322 6607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b55db1ddc34f209ac28fe8639a9a08. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } }
I20260620 23:29:06.344296 6606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:06.344408 6607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:06.345659 6615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:06.353587 6615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:06.363200 6615 catalog_manager.cc:1347] Generated new cluster ID: 0403415ccf4e4c6fa5598b2f045fb462
I20260620 23:29:06.363359 6615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:06.385914 6615 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:06.388134 6615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:06.401612 6615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Generated new TSK 0
I20260620 23:29:06.403038 6615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab.
Entry for principal oryx/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab.
I20260620 23:29:06.478251 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45755
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:06.851814 6626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:06.852115 6626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:06.852253 6626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:06.860008 6626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:06.860132 6626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:06.860319 6626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:06.870330 6626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:06.871721 6626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:06.873098 6626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:06.887002 6631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:06.886969 6632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:06.887230 6634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:06.888429 6626 server_base.cc:1034] running on GCE node
I20260620 23:29:06.889222 6626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:06.890282 6626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:06.891505 6626 hybrid_clock.cc:648] HybridClock initialized: now 1781998146891440 us; error 61 us; skew 500 ppm
I20260620 23:29:06.891855 6626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:06 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:06.895378 6626 init.cc:377] Logged in from keytab as oryx/127.0.96.193@KRBTEST.COM (short username oryx)
I20260620 23:29:06.897194 6626 webserver.cc:460] Webserver started at http://127.0.96.193:43141/ using document root <none> and password file <none>
I20260620 23:29:06.897881 6626 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:06.897987 6626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:06.898276 6626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:06.901396 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "1d9735cd69864afba13d4b00015f484c"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "e5804d8c192c33c8e884b965352bc7f5"
server_key_iv: "81c09a5d65b0731afab42b98c6b27fb2"
server_key_version: "encryptionkey@0"
I20260620 23:29:06.902247 6626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "1d9735cd69864afba13d4b00015f484c"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "e5804d8c192c33c8e884b965352bc7f5"
server_key_iv: "81c09a5d65b0731afab42b98c6b27fb2"
server_key_version: "encryptionkey@0"
I20260620 23:29:06.908031 6626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:29:06.912226 6641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:06.913856 6626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 23:29:06.914065 6626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "1d9735cd69864afba13d4b00015f484c"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "e5804d8c192c33c8e884b965352bc7f5"
server_key_iv: "81c09a5d65b0731afab42b98c6b27fb2"
server_key_version: "encryptionkey@0"
I20260620 23:29:06.914280 6626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:06.942478 6626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:06.947122 6626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:06.947427 6626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:06.948450 6626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:06.950021 6626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:06.950136 6626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:06.950306 6626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:06.950381 6626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:06.989826 6626 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:36653
I20260620 23:29:06.989869 6753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:36653 every 8 connection(s)
I20260620 23:29:06.991542 6626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:06.999279 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6626
I20260620 23:29:06.999536 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:07.000204 387 external_mini_cluster.cc:1428] Setting key cfaa67a6330619e2c2ae934f1f01eddf
Jun 20 23:29:07 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:07.016512 6756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:06.994671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:46055 (local address 127.0.96.254:45755)
0620 23:29:06.995276 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:06.995286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:06.996323 (+ 1037us) server_negotiation.cc:407] Connection header received
0620 23:29:06.998519 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:06.998524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:06.998617 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:06.998748 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:07.000901 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.001908 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.004036 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.004834 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.005056 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:07.009779 (+ 4723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:07.009785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:07.009789 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:07.009834 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:07.012937 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:07.014003 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:07.014009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:07.014012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:07.014150 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:07.014616 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:07.014622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:07.014625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:07.014934 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:07.015101 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:07.015887 (+ 786us) server_negotiation.cc:299] Negotiation successful
0620 23:29:07.016126 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":303,"threads_started":1}
I20260620 23:29:07.017729 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:06.995126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:46055)
0620 23:29:06.996091 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:06.996144 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:06.998159 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:06.999026 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:06.999037 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:06.999826 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:07.000646 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:07.000670 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.002145 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.002150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:07.003816 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:07.003833 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.005227 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.005233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:07.005522 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:07.006492 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:07.006522 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:07.009386 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:07.013250 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:07.013269 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:07.013286 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:07.013850 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:07.014273 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:07.014278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:07.014282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:07.014492 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:07.015087 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:07.015096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:07.015491 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 23:29:07.015800 (+ 309us) client_negotiation.cc:240] Negotiation successful
0620 23:29:07.016387 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":379,"threads_started":1}
I20260620 23:29:07.019945 6754 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45755
I20260620 23:29:07.020488 6754 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:07.021757 6754 heartbeater.cc:511] Master 127.0.96.254:45755 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:29:07.024664 6565 ts_manager.cc:194] Registered new tserver with Master: 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653)
I20260620 23:29:07.026448 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:46055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab.
Entry for principal oryx/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab.
I20260620 23:29:07.055891 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45755
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:07.465801 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:07.466114 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:07.466279 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:07.474242 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:07.474375 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:07.474563 6760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:07.485478 6760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:07.486919 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:07.488356 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:07.505501 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:07.505570 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:07.505594 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:07.506696 6760 server_base.cc:1034] running on GCE node
I20260620 23:29:07.507484 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:07.508661 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:07.509929 6760 hybrid_clock.cc:648] HybridClock initialized: now 1781998147509843 us; error 91 us; skew 500 ppm
I20260620 23:29:07.510310 6760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:07 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998147, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:07.514178 6760 init.cc:377] Logged in from keytab as oryx/127.0.96.194@KRBTEST.COM (short username oryx)
I20260620 23:29:07.516639 6760 webserver.cc:460] Webserver started at http://127.0.96.194:44287/ using document root <none> and password file <none>
I20260620 23:29:07.517500 6760 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:07.517624 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:07.517906 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:07.521240 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "d39b41dad4494e27a2076c4c322db8d5"
format_stamp: "Formatted at 2026-06-20 23:29:07 on dist-test-slave-v34r"
server_key: "f2d091a25ffee5f4fbd0bdbf7975b968"
server_key_iv: "52070552ad672ebb26c91fce2b295653"
server_key_version: "encryptionkey@0"
I20260620 23:29:07.522035 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "d39b41dad4494e27a2076c4c322db8d5"
format_stamp: "Formatted at 2026-06-20 23:29:07 on dist-test-slave-v34r"
server_key: "f2d091a25ffee5f4fbd0bdbf7975b968"
server_key_iv: "52070552ad672ebb26c91fce2b295653"
server_key_version: "encryptionkey@0"
I20260620 23:29:07.528407 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:29:07.532763 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:07.534830 6760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:07.535084 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "d39b41dad4494e27a2076c4c322db8d5"
format_stamp: "Formatted at 2026-06-20 23:29:07 on dist-test-slave-v34r"
server_key: "f2d091a25ffee5f4fbd0bdbf7975b968"
server_key_iv: "52070552ad672ebb26c91fce2b295653"
server_key_version: "encryptionkey@0"
I20260620 23:29:07.535465 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:07.572129 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:07.577633 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:07.577991 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:07.579099 6760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:07.580592 6760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:07.580704 6760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:07.580854 6760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:07.580935 6760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:07.621907 6760 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:43425
I20260620 23:29:07.622009 6887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:43425 every 8 connection(s)
I20260620 23:29:07.623414 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:07.624760 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6760
I20260620 23:29:07.625041 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:07.625847 387 external_mini_cluster.cc:1428] Setting key d8fabb8875d4cfded1fa9795535f9342
Jun 20 23:29:07 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998147, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:07.651507 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:07.627143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:52031 (local address 127.0.96.254:45755)
0620 23:29:07.627837 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:07.627845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:07.629276 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 23:29:07.631921 (+ 2645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:07.631928 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:07.632056 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:07.632194 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:07.634366 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.635241 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.637266 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.638203 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.638415 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:07.642848 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:07.642859 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:07.642868 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:07.642933 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:07.647198 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:07.648256 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:07.648263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:07.648268 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:07.648512 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:07.649084 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:07.649093 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:07.649099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:07.649605 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:07.649788 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:07.650663 (+ 875us) server_negotiation.cc:299] Negotiation successful
0620 23:29:07.650973 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":420,"threads_started":1}
WARNING: no policy specified for oryx/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:29:07.652326 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:07.627867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:52031)
0620 23:29:07.629048 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:07.629122 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:07.631507 (+ 2385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:07.632564 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:07.632583 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:07.633340 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:07.634098 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:07.634121 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.635509 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.635515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:07.636968 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:07.637051 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:07.638556 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:07.638564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:07.638879 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:07.639777 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:07.639820 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:07.642416 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:07.647446 (+ 5030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:07.647458 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:07.647481 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:07.648037 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:07.648680 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:07.648684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:07.648687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:07.648895 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:07.649760 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:07.649766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:07.650201 (+ 435us) client_negotiation.cc:769] Sending connection context
0620 23:29:07.650491 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 23:29:07.650925 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":444,"threads_started":1}
I20260620 23:29:07.654320 6888 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45755
I20260620 23:29:07.654776 6888 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.195@KRBTEST.COM" created.
I20260620 23:29:07.655789 6888 heartbeater.cc:511] Master 127.0.96.254:45755 requested a full tablet report, sending...
I20260620 23:29:07.657908 6565 ts_manager.cc:194] Registered new tserver with Master: d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425)
I20260620 23:29:07.659200 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:52031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab.
Entry for principal oryx/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab.
I20260620 23:29:07.685439 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45755
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:08.030648 6754 heartbeater.cc:503] Master 127.0.96.254:45755 was elected leader, sending a full tablet report...
W20260620 23:29:08.111750 6894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:08.112176 6894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:08.112346 6894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:08.121146 6894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:08.121302 6894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:08.121548 6894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:08.134200 6894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:08.136067 6894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:08.137857 6894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:08.154816 6899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:08.154826 6900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:08.155010 6902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:08.156951 6894 server_base.cc:1034] running on GCE node
I20260620 23:29:08.157874 6894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:08.159200 6894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:08.160465 6894 hybrid_clock.cc:648] HybridClock initialized: now 1781998148160403 us; error 60 us; skew 500 ppm
I20260620 23:29:08.160877 6894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:08 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998148, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:08.164741 6894 init.cc:377] Logged in from keytab as oryx/127.0.96.195@KRBTEST.COM (short username oryx)
I20260620 23:29:08.166852 6894 webserver.cc:460] Webserver started at http://127.0.96.195:46117/ using document root <none> and password file <none>
I20260620 23:29:08.167827 6894 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:08.168007 6894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:08.168398 6894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:08.172219 6894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "4ddded06fcea475ea96243bb51197b85"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "b7b919839f8a72bf9fac5a1575b13205"
server_key_iv: "b0eecb8f70c3bea45c0bc987498a5094"
server_key_version: "encryptionkey@0"
I20260620 23:29:08.173338 6894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "4ddded06fcea475ea96243bb51197b85"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "b7b919839f8a72bf9fac5a1575b13205"
server_key_iv: "b0eecb8f70c3bea45c0bc987498a5094"
server_key_version: "encryptionkey@0"
I20260620 23:29:08.180342 6894 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260620 23:29:08.185359 6909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:08.187393 6894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:08.187700 6894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "4ddded06fcea475ea96243bb51197b85"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "b7b919839f8a72bf9fac5a1575b13205"
server_key_iv: "b0eecb8f70c3bea45c0bc987498a5094"
server_key_version: "encryptionkey@0"
I20260620 23:29:08.188023 6894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:08.205899 6894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:08.211365 6894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:08.211722 6894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:08.213057 6894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:08.214696 6894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:08.214808 6894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:08.214982 6894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:08.215065 6894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:08.257313 6894 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:34527
I20260620 23:29:08.257364 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:34527 every 8 connection(s)
I20260620 23:29:08.258891 6894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:08.264405 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 6894
I20260620 23:29:08.264605 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:08.265201 387 external_mini_cluster.cc:1428] Setting key 9d9333a9b5a05895b586703f5f9b182f
Jun 20 23:29:08 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998148, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:08.282001 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.262049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:34395 (local address 127.0.96.254:45755)
0620 23:29:08.262583 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:08.262588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:08.263852 (+ 1264us) server_negotiation.cc:407] Connection header received
0620 23:29:08.266593 (+ 2741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:08.266600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:08.266740 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:08.266915 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:08.268943 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.269856 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.271486 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.271907 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.272020 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.275439 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:08.275444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:08.275448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:08.275494 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:08.278386 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.279083 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.279087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.279090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.279222 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.279679 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.279685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.279691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.280077 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:08.280250 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:08.281255 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0620 23:29:08.281538 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":263,"threads_started":1}
I20260620 23:29:08.282827 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.262770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:34395)
0620 23:29:08.263622 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:08.263670 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:08.266236 (+ 2566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:08.267257 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:08.267271 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:08.267979 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:08.268735 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.268756 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.270021 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.270025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.271337 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.271348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.272112 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.272116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.272305 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.273048 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:08.273077 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:08.275181 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:08.278555 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.278562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.278573 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.278971 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.279336 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.279341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.279344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.279561 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.280286 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:08.280297 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:08.280750 (+ 453us) client_negotiation.cc:769] Sending connection context
0620 23:29:08.281116 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 23:29:08.281598 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":335,"threads_started":1}
I20260620 23:29:08.284730 7022 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45755
I20260620 23:29:08.285167 7022 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:08.285931 7022 heartbeater.cc:511] Master 127.0.96.254:45755 requested a full tablet report, sending...
I20260620 23:29:08.287533 6565 ts_manager.cc:194] Registered new tserver with Master: 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527)
I20260620 23:29:08.288357 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:34395
I20260620 23:29:08.294902 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.253.keytab.
Entry for principal oryx/127.0.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.253.keytab.
I20260620 23:29:08.348759 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:34031
--webserver_interface=127.0.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.253.keytab
--principal=oryx/127.0.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:45755,127.0.96.253:34031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:08.663349 6888 heartbeater.cc:503] Master 127.0.96.254:45755 was elected leader, sending a full tablet report...
W20260620 23:29:08.711541 7028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:08.711862 7028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:08.711951 7028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:08.719408 7028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:08.719537 7028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:08.719583 7028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:08.719631 7028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:08.719668 7028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:08.729519 7028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:45755,127.0.96.253:34031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.253.keytab
--principal=oryx/127.0.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:34031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
W20260620 23:29:08.729894 7028 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.96.254:45755,127.0.96.253:34031'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 23:29:08.730939 7028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:08.732331 7028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:08.746109 7033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:08.746044 7036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:08.745997 7034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:08.747783 7028 server_base.cc:1034] running on GCE node
I20260620 23:29:08.748905 7028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:08.750849 7028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:08.752177 7028 hybrid_clock.cc:648] HybridClock initialized: now 1781998148752092 us; error 52 us; skew 500 ppm
I20260620 23:29:08.752638 7028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:08 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998148, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:08.756744 7028 init.cc:377] Logged in from keytab as oryx/127.0.96.253@KRBTEST.COM (short username oryx)
I20260620 23:29:08.758826 7028 webserver.cc:460] Webserver started at http://127.0.96.253:44825/ using document root <none> and password file <none>
I20260620 23:29:08.759766 7028 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:08.759927 7028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:08.760324 7028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:08.764170 7028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/instance:
uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "dd8babf827a5e72d8cd53eaf472aded5"
server_key_iv: "be77a6468f2085cc0333352cc083f650"
server_key_version: "encryptionkey@0"
I20260620 23:29:08.765250 7028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal/instance:
uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "dd8babf827a5e72d8cd53eaf472aded5"
server_key_iv: "be77a6468f2085cc0333352cc083f650"
server_key_version: "encryptionkey@0"
I20260620 23:29:08.772441 7028 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 23:29:08.777238 7043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:08.779045 7028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:29:08.779317 7028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "dd8babf827a5e72d8cd53eaf472aded5"
server_key_iv: "be77a6468f2085cc0333352cc083f650"
server_key_version: "encryptionkey@0"
I20260620 23:29:08.779604 7028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:08.803507 7028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:08.809393 7028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:08.809743 7028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:08.832695 7028 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.253:34031
I20260620 23:29:08.832742 7094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.253:34031 every 8 connection(s)
I20260620 23:29:08.834508 7028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
I20260620 23:29:08.838553 7095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:08.844413 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 7028
I20260620 23:29:08.845831 7095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:29:08 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998148, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.253@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
Jun 20 23:29:08 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
I20260620 23:29:08.869565 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.849105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32824 (local address 127.0.96.254:45755)
0620 23:29:08.849666 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:08.849674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:08.850315 (+ 641us) server_negotiation.cc:407] Connection header received
0620 23:29:08.851520 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:08.851525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:08.851627 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:08.851765 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:08.853559 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.854502 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.856467 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.856925 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.857177 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.861514 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:08.861519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:08.861522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:08.861568 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:08.865202 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.866454 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.866459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.866461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.866639 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.867273 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.867278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.867281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.867763 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:08.867963 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:08.868912 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 23:29:08.869208 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":291,"threads_started":1}
I20260620 23:29:08.870962 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.847348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.0.96.253:34031)
0620 23:29:08.848347 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:08.848360 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:08.848452 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:29:08.851175 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:08.851201 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:08.851531 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:08.851808 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:08.853349 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.854763 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.856129 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.856879 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.857128 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.862172 (+ 5044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:08.862190 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:08.862214 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:08.862271 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:08.866214 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.867273 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.867282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.867285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.867446 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.867858 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.867863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.867866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.868394 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:08.868629 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:08.869163 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 23:29:08.869690 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":206,"server-negotiator.queue_time_us":734,"thread_start_us":477,"threads_started":1}
I20260620 23:29:08.871074 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.849193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:32824)
0620 23:29:08.850142 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:08.850190 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:08.851215 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:08.851856 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:08.851863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:08.852773 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:08.853341 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.853359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.854699 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.854706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.856281 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.856293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.857299 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.857312 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.857471 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.858269 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:08.858294 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:08.861253 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:08.865398 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.865412 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.865427 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.866259 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.866767 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.866774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.866780 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.867072 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.867935 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:08.867943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:08.868411 (+ 468us) client_negotiation.cc:769] Sending connection context
0620 23:29:08.868788 (+ 377us) client_negotiation.cc:240] Negotiation successful
0620 23:29:08.869495 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":494,"threads_started":1}
I20260620 23:29:08.873355 7095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:29:08 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998148, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.253@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
I20260620 23:29:08.887652 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.874047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48240)
0620 23:29:08.874253 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:08.874274 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:08.874389 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:08.875159 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:08.875163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:08.875542 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:08.875753 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.875761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.876827 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.876832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.877861 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.877872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.878595 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.878599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.878765 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.879369 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:08.879390 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:08.881752 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:08.885297 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.885301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.885304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.885724 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.886204 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.886210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.886213 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.886349 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.886886 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:08.886889 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:08.887030 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 23:29:08.887168 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:29:08.887397 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:29:08.887975 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.874608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.96.253:34031)
0620 23:29:08.874832 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:08.874841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:08.874865 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:29:08.874927 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:08.874931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:08.874993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:08.875102 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:08.875883 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.876651 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.877999 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.878449 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.878575 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.882001 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:08.882007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:08.882011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:08.882052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:08.885103 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.885851 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.885856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.885859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.886017 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.886455 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.886458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.886460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.886776 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:08.886897 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:08.887254 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 23:29:08.887557 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":65}
I20260620 23:29:08.893880 7095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Bootstrap starting.
I20260620 23:29:08.897992 7095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:08.899282 7095 log.cc:824] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:08.902722 7095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: No bootstrap required, opened a new log
I20260620 23:29:08.910102 7095 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } }
I20260620 23:29:08.910465 7095 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:08.910596 7095 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd4a7787ced42269d1b4f3d43cd9772, State: Initialized, Role: FOLLOWER
I20260620 23:29:08.911270 7095 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } }
I20260620 23:29:08.912353 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } } }
I20260620 23:29:08.912768 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:08.913463 7095 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:29:08.928414 7113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:29:08.928704 7113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:29:08.932291 7028 master_runner.cc:428] Detected that this master ffd4a7787ced42269d1b4f3d43cd9772 is joining an existing cluster
I20260620 23:29:08.932431 7028 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.96.253:34031
WARNING: no policy specified for oryx/127.0.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.252@KRBTEST.COM" created.
I20260620 23:29:08.981781 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.955110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.0.96.254:45755)
0620 23:29:08.955301 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:08.955307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:08.955664 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:29:08.955734 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:08.955739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:08.955837 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:08.955932 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:08.965041 (+ 9109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.966265 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.969264 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.970128 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.970257 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.973448 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:08.973457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:08.973465 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:08.973512 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:08.977818 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.978909 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.978918 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.978924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.979108 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.979647 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.979655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.979660 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.980112 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:08.980286 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:08.981047 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 23:29:08.981334 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:29:08.983590 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.954380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:32828)
0620 23:29:08.955302 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:08.955323 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:08.955495 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:08.962454 (+ 6959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:08.962462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:08.964196 (+ 1734us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:08.964738 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.964755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.966514 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.966519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.969025 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.969045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.970746 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.970756 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.971004 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.971755 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:08.971783 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:08.973208 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:08.978058 (+ 4850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.978066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.978073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.978709 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.979282 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.979288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.979293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.979488 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.980380 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:08.980388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:08.980643 (+ 255us) client_negotiation.cc:769] Sending connection context
0620 23:29:08.982429 (+ 1786us) client_negotiation.cc:240] Negotiation successful
0620 23:29:08.982887 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":2,"thread_start_us":920,"threads_started":1}
I20260620 23:29:08.999589 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.956808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48256)
0620 23:29:08.964240 (+ 7432us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:08.964268 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:08.964540 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:08.965135 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:08.965141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:08.965834 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:08.966201 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.966214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.967831 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.967835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.969768 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:08.969788 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.983521 (+ 13733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.983529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:08.983754 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.984535 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:08.984564 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:08.985839 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:08.996100 (+ 10261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.996108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.996113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.996816 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.997432 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:08.997438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:08.997442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:08.997606 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:08.998305 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:08.998310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:08.998456 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 23:29:08.998605 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 23:29:08.999015 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7230,"thread_start_us":6879,"threads_started":1}
I20260620 23:29:09.001034 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.957474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.96.253:34031)
0620 23:29:08.957758 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:08.957766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:08.964407 (+ 6641us) server_negotiation.cc:407] Connection header received
0620 23:29:08.964716 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:08.964723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:08.964797 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:08.964880 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:08.966365 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.967563 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.982044 (+ 14481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:08.982935 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:08.983086 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:08.986106 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:08.986117 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:08.986123 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:08.986160 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:08.995868 (+ 9708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.997101 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.997107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.997111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.997301 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:08.997733 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:08.997738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:08.997741 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:08.998172 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.000154 (+ 1982us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.000308 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.000598 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.254@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.252.keytab.
Entry for principal oryx/127.0.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.252.keytab.
I20260620 23:29:09.026938 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:34727
--webserver_interface=127.0.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.252.keytab
--principal=oryx/127.0.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:09.027732 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.991394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48268)
0620 23:29:09.003333 (+ 11939us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.003406 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.003917 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.005049 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.005061 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.006193 (+ 1132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.006588 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.006607 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.008606 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.008617 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.010738 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.010763 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.012283 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.012295 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.012611 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.013799 (+ 1188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.013831 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.017783 (+ 3952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.023663 (+ 5880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.023684 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.023692 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.024410 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.024949 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.024955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.024959 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.025256 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.026191 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.026205 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.026506 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.026813 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.027281 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11550,"thread_start_us":11352,"threads_started":1}
I20260620 23:29:09.029223 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:08.991714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.96.253:34031)
0620 23:29:09.001329 (+ 9615us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.001336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.003536 (+ 2200us) server_negotiation.cc:407] Connection header received
0620 23:29:09.004276 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.004285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.004422 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.004534 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.006843 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.008338 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.011067 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.011987 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.012180 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.018194 (+ 6014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.018202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.018206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.018262 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.022958 (+ 4696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.024591 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.024595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.024598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.024794 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.025410 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.025435 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.025445 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.025899 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.026101 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.028357 (+ 2256us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.028629 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9494,"thread_start_us":12988,"threads_started":1}
I20260620 23:29:09.031328 6565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.96.253:34031: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
member_type: NON_VOTER
last_known_addr {
host: "127.0.96.253"
port: 34031
}
attrs {
promote: true
}
}
dest_uuid: "36b55db1ddc34f209ac28fe8639a9a08"
cas_config_opid_index: -1
I20260620 23:29:09.032779 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } }
I20260620 23:29:09.041265 7127 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.042160 7070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:09.047179 7070 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Refusing update from remote peer 36b55db1ddc34f209ac28fe8639a9a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 23:29:09.050675 7126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:29:09.062402 7049 tablet.cc:2369] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:09.065287 7127 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.96.253:34031
I20260620 23:29:09.066725 7028 master.cc:537] Master@127.0.96.253:34031 shutting down...
I20260620 23:29:09.067616 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.067994 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:09.087116 7070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.106045 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.106570 7102 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b55db1ddc34f209ac28fe8639a9a08. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.106900 7102 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:09.107177 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.107544 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:09.106842 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:09.111277 7028 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 LEARNER]: Raft consensus shutting down.
I20260620 23:29:09.111858 7028 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 LEARNER]: Raft consensus is shut down!
I20260620 23:29:09.112046 7028 tablet_replica.cc:330] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: stopping tablet replica
I20260620 23:29:09.121388 7028 master.cc:552] Master@127.0.96.253:34031 shutdown complete.
I20260620 23:29:09.121515 7028 master_runner.cc:305] Clearing existing system tablet
I20260620 23:29:09.122041 7028 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:09.122148 7028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:09.125584 7028 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 23:29:09.129402 7140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:09.129974 7028 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:09.130288 7028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:29:09.130388 7028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "dd8babf827a5e72d8cd53eaf472aded5"
server_key_iv: "be77a6468f2085cc0333352cc083f650"
server_key_version: "encryptionkey@0"
I20260620 23:29:09.135593 7028 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:09.144009 7028 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:29:09.144259 7028 log.cc:1196] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 23:29:09.144809 7028 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Deleting consensus metadata
I20260620 23:29:09.145164 7028 master_runner.cc:315] Copying system tablet from 127.0.96.254:45755
I20260620 23:29:09.150846 7028 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.254:45755
I20260620 23:29:09.170315 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.151951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32842 (local address 127.0.96.254:45755)
0620 23:29:09.152201 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.152207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.153721 (+ 1514us) server_negotiation.cc:407] Connection header received
0620 23:29:09.153999 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.154006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.154128 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.154264 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.155726 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.156877 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.159077 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.159781 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.159925 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.162522 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.162528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.162533 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.162590 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.166527 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.167615 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.167623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.167627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.167845 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.168362 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.168367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.168371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.168764 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.168928 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.169613 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.169889 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:29:09.170504 7146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.151847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:32842)
0620 23:29:09.153519 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.153556 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.153781 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.154341 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.154346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.155047 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.155482 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.155500 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.157126 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.157136 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.158849 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.158864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.159970 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.159978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.160247 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.161083 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.161116 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.162247 (+ 1131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.166806 (+ 4559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.166812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.166817 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.167412 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.168002 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.168008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.168012 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.168221 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.168910 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.168915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.169232 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.169463 (+ 231us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.169980 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"thread_start_us":1346,"threads_started":1}
I20260620 23:29:09.171798 6585 tablet_copy_service.cc:140] P 36b55db1ddc34f209ac28fe8639a9a08: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ffd4a7787ced42269d1b4f3d43cd9772 ({username='oryx', principal='oryx/127.0.96.253@KRBTEST.COM'} at 127.0.0.1:32842)
I20260620 23:29:09.172142 6585 tablet_copy_service.cc:161] P 36b55db1ddc34f209ac28fe8639a9a08: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ffd4a7787ced42269d1b4f3d43cd9772 at {username='oryx', principal='oryx/127.0.96.253@KRBTEST.COM'} at 127.0.0.1:32842: session id = ffd4a7787ced42269d1b4f3d43cd9772-00000000000000000000000000000000
I20260620 23:29:09.176837 6585 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:29:09.179678 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:29:09.187866 7028 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: tablet copy: Starting download of 0 data blocks...
I20260620 23:29:09.188272 7028 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: tablet copy: Starting download of 1 WAL segments...
I20260620 23:29:09.192240 7028 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:29:09.198125 6585 tablet_copy_service.cc:342] P 36b55db1ddc34f209ac28fe8639a9a08: Request end of tablet copy session ffd4a7787ced42269d1b4f3d43cd9772-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.96.253@KRBTEST.COM'} at 127.0.0.1:32842
I20260620 23:29:09.198383 6585 tablet_copy_service.cc:433] P 36b55db1ddc34f209ac28fe8639a9a08: ending tablet copy session ffd4a7787ced42269d1b4f3d43cd9772-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ffd4a7787ced42269d1b4f3d43cd9772
I20260620 23:29:09.213327 7028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:09.216454 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:09.221349 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:09.226639 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:09.226578 7028 server_base.cc:1034] running on GCE node
I20260620 23:29:09.227449 7028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:09.228237 7028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:09.229473 7028 hybrid_clock.cc:648] HybridClock initialized: now 1781998149229392 us; error 60 us; skew 500 ppm
I20260620 23:29:09.229717 7028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:09.232558 7028 init.cc:377] Logged in from keytab as oryx/127.0.96.253@KRBTEST.COM (short username oryx)
I20260620 23:29:09.238138 7028 webserver.cc:460] Webserver started at http://127.0.96.253:33061/ using document root <none> and password file <none>
I20260620 23:29:09.238621 7028 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:09.238775 7028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:09.242648 7028 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:29:09.245061 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:09.245586 7028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 23:29:09.245788 7028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "dd8babf827a5e72d8cd53eaf472aded5"
server_key_iv: "be77a6468f2085cc0333352cc083f650"
server_key_version: "encryptionkey@0"
I20260620 23:29:09.246016 7028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:09.284904 7028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:09.289628 7028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:09.291326 7022 heartbeater.cc:503] Master 127.0.96.254:45755 was elected leader, sending a full tablet report...
I20260620 23:29:09.344749 7028 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.253:34031
I20260620 23:29:09.344776 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.253:34031 every 8 connection(s)
I20260620 23:29:09.345999 7028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
I20260620 23:29:09.353246 7209 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:29:09.359190 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Bootstrap starting.
I20260620 23:29:09.370307 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:29:09.371061 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Bootstrap complete.
I20260620 23:29:09.372051 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } }
I20260620 23:29:09.372280 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ffd4a7787ced42269d1b4f3d43cd9772, State: Initialized, Role: LEARNER
I20260620 23:29:09.372515 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } }
I20260620 23:29:09.373364 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.373714 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:09.373852 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:09.386301 7223 catalog_manager.cc:1259] Loaded cluster ID: 0403415ccf4e4c6fa5598b2f045fb462
I20260620 23:29:09.386413 7223 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: loading cluster ID for follower catalog manager: success
I20260620 23:29:09.390056 7223 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: acquiring CA information for follower catalog manager: success
I20260620 23:29:09.392599 7223 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:29:09.526292 7128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:09.526602 7128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:09.526700 7128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:09.534456 7128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:09.534572 7128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:09.534618 7128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:09.534667 7128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:09.534723 7128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:09.546017 7128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.252.keytab
--principal=oryx/127.0.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:09.547564 7128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:09.548877 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:09.563068 7231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:09.563186 7229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:09.563175 7228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:09.570720 7128 server_base.cc:1034] running on GCE node
I20260620 23:29:09.571722 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:09.573429 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:09.574664 7128 hybrid_clock.cc:648] HybridClock initialized: now 1781998149574581 us; error 79 us; skew 500 ppm
I20260620 23:29:09.575083 7128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:09.578540 7128 init.cc:377] Logged in from keytab as oryx/127.0.96.252@KRBTEST.COM (short username oryx)
I20260620 23:29:09.580263 7128 webserver.cc:460] Webserver started at http://127.0.96.252:40441/ using document root <none> and password file <none>
I20260620 23:29:09.581072 7128 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:09.581182 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:09.581446 7128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:09.584622 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/instance:
uuid: "643e504e8b2d4db28aeee4b200eb73eb"
format_stamp: "Formatted at 2026-06-20 23:29:09 on dist-test-slave-v34r"
server_key: "42fff121b0d39be41de3bd97f2675cdf"
server_key_iv: "2d01a217a82ffa18a263821227bd719a"
server_key_version: "encryptionkey@0"
I20260620 23:29:09.585417 7128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal/instance:
uuid: "643e504e8b2d4db28aeee4b200eb73eb"
format_stamp: "Formatted at 2026-06-20 23:29:09 on dist-test-slave-v34r"
server_key: "42fff121b0d39be41de3bd97f2675cdf"
server_key_iv: "2d01a217a82ffa18a263821227bd719a"
server_key_version: "encryptionkey@0"
I20260620 23:29:09.591046 7128 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:29:09.595450 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:09.597213 7128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:29:09.597414 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
uuid: "643e504e8b2d4db28aeee4b200eb73eb"
format_stamp: "Formatted at 2026-06-20 23:29:09 on dist-test-slave-v34r"
server_key: "42fff121b0d39be41de3bd97f2675cdf"
server_key_iv: "2d01a217a82ffa18a263821227bd719a"
server_key_version: "encryptionkey@0"
I20260620 23:29:09.597620 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:09.629416 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:09.631655 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.632017 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:09.635943 7128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:09.636305 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:09.643482 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.632408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.96.253:34031)
0620 23:29:09.635264 (+ 2856us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.635271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.635294 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:29:09.635378 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.635384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.635482 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.635630 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:09.636333 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.638278 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.640949 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.641936 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.642085 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.642381 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.642682 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.643047 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2711,"thread_start_us":2540,"threads_started":1}
I20260620 23:29:09.643800 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.631849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48278)
0620 23:29:09.633374 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.633397 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.633571 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.635854 (+ 2283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.635859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.635887 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:09.636096 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.636111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.638525 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.638532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.640719 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.640738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.642149 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.642165 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.642412 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.642499 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.642622 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.642820 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":1074,"threads_started":1}
I20260620 23:29:09.645236 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b55db1ddc34f209ac28fe8639a9a08. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.645644 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:09.646562 7243 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: attempting to promote NON_VOTER ffd4a7787ced42269d1b4f3d43cd9772 to VOTER
I20260620 23:29:09.647571 7243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } }
I20260620 23:29:09.649899 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 LEARNER]: Refusing update from remote peer 36b55db1ddc34f209ac28fe8639a9a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 23:29:09.650681 7244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 23:29:09.659404 7244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } }
I20260620 23:29:09.660229 7184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } }
I20260620 23:29:09.664350 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } }
I20260620 23:29:09.664630 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:09.665154 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } }
I20260620 23:29:09.665556 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:09.666348 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } }
I20260620 23:29:09.666559 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:09.671502 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.252:34727
I20260620 23:29:09.671525 7296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.252:34727 every 8 connection(s)
I20260620 23:29:09.673266 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
I20260620 23:29:09.674100 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 7128
I20260620 23:29:09.679913 7297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:09.689806 7297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.252@KRBTEST.COM
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.252@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:09.707752 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.676962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.0.96.252:34727)
0620 23:29:09.680971 (+ 4009us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.681048 (+ 77us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.681695 (+ 647us) server_negotiation.cc:407] Connection header received
0620 23:29:09.684444 (+ 2749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.684479 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.685187 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.685742 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.687539 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.689249 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.690674 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.691504 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.691730 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.698170 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.698180 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.698197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.698255 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.702686 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.703635 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.703650 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.703663 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.703862 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.704277 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.704282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.704286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.704967 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.705366 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.705878 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.706407 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3682,"thread_start_us":3365,"threads_started":1}
I20260620 23:29:09.710278 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.692804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.0.96.254:45755)
0620 23:29:09.694177 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.694182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.694197 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:29:09.694248 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.694252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.694314 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.694449 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.695606 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.696772 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.698434 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.699017 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.699185 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.704082 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.704088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.704091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.704127 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.706983 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.707871 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.707876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.707879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.708047 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.708524 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.708531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.708535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.708947 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.709160 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.709747 (+ 587us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.709971 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":1119,"threads_started":1}
I20260620 23:29:09.710469 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.692664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:32848)
0620 23:29:09.693529 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.693573 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.693752 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.694518 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.694526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.695174 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.695433 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.695442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.696929 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.696934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.698128 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.698142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.699284 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.699292 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.699579 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.700503 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.700532 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.703852 (+ 3320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.707134 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.707153 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.707162 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.707725 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.708192 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.708196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.708198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.708321 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.709150 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.709157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.709370 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.709624 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.710021 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":449,"threads_started":1}
I20260620 23:29:09.712327 7297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.252@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
I20260620 23:29:09.726833 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.713252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48282)
0620 23:29:09.713458 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.713481 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.713584 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.713954 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.713958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.714304 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.714446 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.714455 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.716266 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.716270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.717436 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.717451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.718390 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.718395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.718591 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.719310 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.719327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.721434 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.724556 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.724560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.724563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.724969 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.725447 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.725451 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.725453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.725584 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.726100 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.726103 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.726260 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.726373 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.726591 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:29:09.727048 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.713304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.0.96.253:34031)
0620 23:29:09.713513 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.713519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.713538 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:29:09.713712 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.713716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.713787 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.713882 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.714589 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.716089 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.717672 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.718139 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.718250 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.721677 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.721683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.721687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.721731 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.724400 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.725167 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.725171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.725174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.725337 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.725699 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.725701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.725703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.725988 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.726115 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.726483 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.726703 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:29:09.728255 7297 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998149, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.252@KRBTEST.COM for oryx/127.0.96.252@KRBTEST.COM
I20260620 23:29:09.741858 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.728931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.0.1:55498)
0620 23:29:09.729162 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.729180 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.729280 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.729640 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.729644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.729980 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.730131 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.730138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.731128 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.731132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.732078 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.732090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.732948 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.732952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.733223 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.733876 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.733894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.735920 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.739282 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.739286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.739289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.739744 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.740194 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.740198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.740201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.740367 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.741044 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.741047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.741214 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.741353 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.741547 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 23:29:09.743110 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.729196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.96.252:34727)
0620 23:29:09.729365 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.729369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.729385 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:29:09.729432 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.729435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.729499 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.729597 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.730257 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.730999 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.732229 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.732770 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.732894 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.736162 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.736168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.736172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.736210 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.739114 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.739916 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.739926 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.739929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.740077 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.740488 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.740492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.740494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.740884 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.741066 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.741517 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.741815 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 23:29:09.749387 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Bootstrap starting.
I20260620 23:29:09.753489 7297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:09.754736 7297 log.cc:824] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:09.758424 7297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: No bootstrap required, opened a new log
I20260620 23:29:09.767567 7297 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } }
I20260620 23:29:09.768020 7297 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:09.768115 7297 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643e504e8b2d4db28aeee4b200eb73eb, State: Initialized, Role: FOLLOWER
I20260620 23:29:09.768872 7297 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } }
I20260620 23:29:09.770068 7304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } } }
I20260620 23:29:09.770519 7304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:09.771517 7297 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:29:09.783561 7315 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:29:09.783795 7315 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:29:09.789758 7128 master_runner.cc:428] Detected that this master 643e504e8b2d4db28aeee4b200eb73eb is joining an existing cluster
I20260620 23:29:09.789965 7128 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.96.252:34727
I20260620 23:29:09.815557 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6626
I20260620 23:29:09.819597 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.796116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.0.96.254:45755)
0620 23:29:09.796361 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.796368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.797193 (+ 825us) server_negotiation.cc:407] Connection header received
0620 23:29:09.797534 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.797542 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.797664 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.797754 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.801579 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.802732 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.805699 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.806422 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.806542 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.810023 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.810032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.810039 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.810103 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.815037 (+ 4934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.816120 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.816127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.816131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.816302 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.816835 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.816843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.816849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.817755 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.818022 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.818790 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.819189 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:29:09.821102 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.796139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:32862)
0620 23:29:09.797045 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.797067 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.797341 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.798033 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.798038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.799908 (+ 1870us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.800440 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.800457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.803411 (+ 2954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.803420 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.805467 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.805482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.806891 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.806910 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.807328 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.808288 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.808315 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.809708 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.815264 (+ 5556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.815270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.815275 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.815942 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.816469 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.816474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.816478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.816698 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.818109 (+ 1411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.818114 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.818362 (+ 248us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.819929 (+ 1567us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.820487 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":136,"spinlock_wait_cycles":38272,"thread_start_us":664,"threads_started":1}
I20260620 23:29:09.824462 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.796745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.96.253:34031)
0620 23:29:09.796973 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.797009 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.799407 (+ 2398us) server_negotiation.cc:407] Connection header received
0620 23:29:09.800874 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.800882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.801019 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.801107 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.803658 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.804816 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.807479 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.808176 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.808285 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.811626 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.811632 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.811636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.811679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.819305 (+ 7626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.821516 (+ 2211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.821526 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.821533 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.821756 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.822405 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.822413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.822419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.822926 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.823111 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.823800 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.824081 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:29:09.826411 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.797990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.0.96.252:34727)
0620 23:29:09.798306 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.798315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.801831 (+ 3516us) server_negotiation.cc:407] Connection header received
0620 23:29:09.802080 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.802085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.802156 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.802230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.804521 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.805731 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.807689 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.808399 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.808492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.811370 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.811376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.811380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.811410 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.815652 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.816498 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.816503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.816506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.816729 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.823797 (+ 7068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.823801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.823804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.824181 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.824308 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.825309 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.825604 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":71}
I20260620 23:29:09.826452 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.796284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48296)
0620 23:29:09.799255 (+ 2971us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.799289 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.800689 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.801379 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.801388 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.802348 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.803385 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.803416 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.805092 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.805097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.807264 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.807278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.808599 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.808604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.808797 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.809804 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.809824 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.811416 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.819557 (+ 8141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.819565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.819572 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.821004 (+ 1432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.822003 (+ 999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.822011 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.822017 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.822256 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.823205 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.823209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.823435 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.825092 (+ 1657us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.825474 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2797,"mutex_wait_us":75,"thread_start_us":2539,"threads_started":1}
I20260620 23:29:09.827502 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.797593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.0.1:55508)
0620 23:29:09.801734 (+ 4141us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.801755 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.801944 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.802420 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.802424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.803958 (+ 1534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.804348 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.804364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.805925 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.805931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.807573 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.807584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.808776 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.808782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.809020 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.809699 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.809733 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.811151 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.815806 (+ 4655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.815810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.815813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.816378 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.823205 (+ 6827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.823209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.823213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.823374 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.824422 (+ 1048us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.824429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.824640 (+ 211us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.825440 (+ 800us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.826979 (+ 1539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3834,"mutex_wait_us":66,"spinlock_wait_cycles":2432,"thread_start_us":3631,"threads_started":1}
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998146, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.254@KRBTEST.COM for oryx/127.0.96.252@KRBTEST.COM
Jun 20 23:29:09 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.193@KRBTEST.COM
I20260620 23:29:09.848414 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.830426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.0.1:55514)
0620 23:29:09.830724 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.830750 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.830911 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.831424 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.831432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.831962 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.832152 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.832162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.833979 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.833988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.835464 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.835487 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.836753 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.836760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.837049 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.837972 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.837999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.840849 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.844855 (+ 4006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.844863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.844868 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.845464 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.845923 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.845929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.845933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.846058 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.846913 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.846919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.847105 (+ 186us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.847295 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.847734 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 23:29:09.849182 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.830636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55514 (local address 127.0.96.252:34727)
0620 23:29:09.830809 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.830816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.830838 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:29:09.831138 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.831144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.831248 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.831369 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.832331 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.833741 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.835786 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.836541 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.837503 (+ 962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.841164 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.841170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.841173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.841211 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.844651 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.845618 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.845622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.845624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.845814 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.846171 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.846178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.846183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.846757 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.846914 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.847602 (+ 688us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.848672 (+ 1070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:29:09.851066 6565 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.96.252:34727: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.96.252"
port: 34727
}
attrs {
promote: true
}
}
dest_uuid: "36b55db1ddc34f209ac28fe8639a9a08"
cas_config_opid_index: 6
I20260620 23:29:09.851909 6565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } }
I20260620 23:29:09.854506 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Refusing update from remote peer 36b55db1ddc34f209ac28fe8639a9a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:29:09.854730 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.817166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.96.193:36653)
0620 23:29:09.831863 (+ 14697us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.831875 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.831911 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:29:09.831982 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.831991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.832565 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.832744 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.834508 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.835552 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.836927 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.837708 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.837860 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.844668 (+ 6808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.844686 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.844710 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.844763 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.849854 (+ 5091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.850640 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.850652 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.850654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.850828 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.851328 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.851332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.851335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.851765 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.853839 (+ 2074us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.854002 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.854317 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14489,"thread_start_us":15064,"threads_started":1}
I20260620 23:29:09.854776 7272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:09.855610 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 23:29:09.858572 7272 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Refusing update from remote peer 36b55db1ddc34f209ac28fe8639a9a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 23:29:09.858683 7244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:09.859687 7184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:09.860931 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:09.861428 6749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:09.863554 7212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:09.863853 7212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:09.864445 7244 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.96.252:34727
I20260620 23:29:09.865237 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:09.865362 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:09.865664 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:09.865748 7128 master.cc:537] Master@127.0.96.252:34727 shutting down...
I20260620 23:29:09.865571 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:09.878525 7271 raft_consensus.cc:858] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } }
W20260620 23:29:09.905365 7250 tablet.cc:2369] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:29:09.878722 7271 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } }
I20260620 23:29:09.919124 7271 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.923578 7331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.924078 7331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 23:29:09.924295 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.924628 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 23:29:09.923974 7304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b55db1ddc34f209ac28fe8639a9a08. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: NON_VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: true } } }
I20260620 23:29:09.926080 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:09.926311 7304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 23:29:09.926393 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
W20260620 23:29:09.927500 6550 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 23:29:09.927891 7128 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 23:29:09.928501 7128 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 23:29:09.928699 7128 tablet_replica.cc:330] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: stopping tablet replica
W20260620 23:29:09.934937 7241 connection.cc:441] server connection from 127.0.0.1:55514 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55514 (request call id 4) could send its response
W20260620 23:29:09.935981 6550 consensus_peers.cc:487] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 -> Peer 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252:34727): Couldn't send request to peer 643e504e8b2d4db28aeee4b200eb73eb. Status: Network error: recv got EOF from 127.0.96.252:34727 (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:29:09.940433 7128 master.cc:552] Master@127.0.96.252:34727 shutdown complete.
I20260620 23:29:09.940630 7128 master_runner.cc:305] Clearing existing system tablet
I20260620 23:29:09.941365 7128 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:09.941574 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:09.944830 7128 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 23:29:09.946657 7334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:09.947201 7128 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:09.947475 7128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260620 23:29:09.947578 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
uuid: "643e504e8b2d4db28aeee4b200eb73eb"
format_stamp: "Formatted at 2026-06-20 23:29:09 on dist-test-slave-v34r"
server_key: "42fff121b0d39be41de3bd97f2675cdf"
server_key_iv: "2d01a217a82ffa18a263821227bd719a"
server_key_version: "encryptionkey@0"
I20260620 23:29:09.950914 7128 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:09.956936 7128 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 23:29:09.957224 7128 log.cc:1196] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 23:29:09.957715 7128 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Deleting consensus metadata
I20260620 23:29:09.957969 7128 master_runner.cc:315] Copying system tablet from 127.0.96.254:45755
I20260620 23:29:09.960803 7128 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.96.254:45755
I20260620 23:29:09.974503 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.961691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.0.96.254:45755)
0620 23:29:09.962033 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:09.962041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:09.962374 (+ 333us) server_negotiation.cc:407] Connection header received
0620 23:29:09.962594 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:09.962600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:09.962684 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:09.962794 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:09.963610 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.964742 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.966117 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.966550 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.966692 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.968549 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:09.968555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:09.968562 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:09.968599 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:09.971322 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.972120 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.972125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.972129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.972265 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:09.972684 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:09.972691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:09.972694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:09.973202 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:09.973348 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:09.973812 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 23:29:09.974078 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260620 23:29:09.974532 7340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:09.961672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:32872)
0620 23:29:09.962243 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:09.962264 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:09.962439 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:09.962819 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:09.962824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:09.963279 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:09.963464 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.963472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.964895 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.964901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.965966 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:09.965977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:09.966712 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:09.966718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:09.966884 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:09.967575 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:09.967593 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:09.968337 (+ 744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:09.971480 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.971485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.971489 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.971994 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.972378 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:09.972383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:09.972387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:09.972567 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:09.973403 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:09.973406 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:09.973543 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 23:29:09.973699 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 23:29:09.974024 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260620 23:29:09.975864 6585 tablet_copy_service.cc:140] P 36b55db1ddc34f209ac28fe8639a9a08: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 643e504e8b2d4db28aeee4b200eb73eb ({username='oryx', principal='oryx/127.0.96.252@KRBTEST.COM'} at 127.0.0.1:32872)
I20260620 23:29:09.976130 6585 tablet_copy_service.cc:161] P 36b55db1ddc34f209ac28fe8639a9a08: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 643e504e8b2d4db28aeee4b200eb73eb at {username='oryx', principal='oryx/127.0.96.252@KRBTEST.COM'} at 127.0.0.1:32872: session id = 643e504e8b2d4db28aeee4b200eb73eb-00000000000000000000000000000000
I20260620 23:29:09.980131 6585 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 23:29:09.982695 7128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 23:29:09.990371 7128 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: tablet copy: Starting download of 0 data blocks...
I20260620 23:29:09.990728 7128 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: tablet copy: Starting download of 1 WAL segments...
I20260620 23:29:09.994437 7128 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 23:29:09.999526 6585 tablet_copy_service.cc:342] P 36b55db1ddc34f209ac28fe8639a9a08: Request end of tablet copy session 643e504e8b2d4db28aeee4b200eb73eb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.96.252@KRBTEST.COM'} at 127.0.0.1:32872
I20260620 23:29:09.999706 6585 tablet_copy_service.cc:433] P 36b55db1ddc34f209ac28fe8639a9a08: ending tablet copy session 643e504e8b2d4db28aeee4b200eb73eb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 643e504e8b2d4db28aeee4b200eb73eb
I20260620 23:29:10.015062 7128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:10.018288 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:10.026643 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:10.037396 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6626
I20260620 23:29:10.053417 7128 server_base.cc:1034] running on GCE node
W20260620 23:29:10.054978 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:10.055660 7128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:10.056550 7128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:10.057785 7128 hybrid_clock.cc:648] HybridClock initialized: now 1781998150057702 us; error 65 us; skew 500 ppm
I20260620 23:29:10.058100 7128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:10 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998150, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:10.060348 7128 init.cc:377] Logged in from keytab as oryx/127.0.96.252@KRBTEST.COM (short username oryx)
I20260620 23:29:10.062107 7128 webserver.cc:460] Webserver started at http://127.0.96.252:41761/ using document root <none> and password file <none>
I20260620 23:29:10.062451 7128 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:10.062568 7128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:10.065793 7128 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 23:29:10.066154 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:36653
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:45755
--webserver_port=43141
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:10.067880 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:10.068444 7128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 23:29:10.068596 7128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
uuid: "643e504e8b2d4db28aeee4b200eb73eb"
format_stamp: "Formatted at 2026-06-20 23:29:09 on dist-test-slave-v34r"
server_key: "42fff121b0d39be41de3bd97f2675cdf"
server_key_iv: "2d01a217a82ffa18a263821227bd719a"
server_key_version: "encryptionkey@0"
I20260620 23:29:10.068799 7128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:10.084806 7128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:10.088863 7128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:10.137125 7128 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.252:34727
I20260620 23:29:10.137151 7403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.252:34727 every 8 connection(s)
I20260620 23:29:10.138309 7128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
I20260620 23:29:10.148891 7404 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:29:10.153929 7404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Bootstrap starting.
I20260620 23:29:10.169862 7404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:29:10.170524 7404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: Bootstrap complete.
I20260620 23:29:10.171620 7404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } }
I20260620 23:29:10.171911 7404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 643e504e8b2d4db28aeee4b200eb73eb, State: Initialized, Role: LEARNER
I20260620 23:29:10.172176 7404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } }
I20260620 23:29:10.173633 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:10.174014 7404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:10.174155 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:10.186028 7418 catalog_manager.cc:1259] Loaded cluster ID: 0403415ccf4e4c6fa5598b2f045fb462
I20260620 23:29:10.186193 7418 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: loading cluster ID for follower catalog manager: success
I20260620 23:29:10.190202 7418 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: acquiring CA information for follower catalog manager: success
I20260620 23:29:10.193859 7418 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:29:10.387636 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:10.388038 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:10.395228 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.387949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.0.1:55524)
0620 23:29:10.388714 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:10.388743 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:10.388953 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:10.389513 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:10.389518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:10.389546 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:10.389825 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.389836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.391211 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.391220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.392653 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.392665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.394250 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.394256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.394445 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.394492 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:29:10.394632 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:29:10.394802 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":438,"threads_started":1}
I20260620 23:29:10.395228 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.388198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.0.96.252:34727)
0620 23:29:10.389130 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:10.389135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:10.389153 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:29:10.389227 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:10.389231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:10.389325 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:10.389498 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:10.390003 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.391062 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.392834 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.394018 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.394177 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.394500 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:10.394660 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:29:10.394802 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":520,"threads_started":1}
I20260620 23:29:10.396430 7379 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260620 23:29:10.396965 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36b55db1ddc34f209ac28fe8639a9a08. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: NON_VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: true } } }
I20260620 23:29:10.397281 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: LEARNER
I20260620 23:29:10.398195 7245 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: attempting to promote NON_VOTER 643e504e8b2d4db28aeee4b200eb73eb to VOTER
I20260620 23:29:10.399073 7245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:10.401486 7379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 LEARNER]: Refusing update from remote peer 36b55db1ddc34f209ac28fe8639a9a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:29:10.401516 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Refusing update from remote peer 36b55db1ddc34f209ac28fe8639a9a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 23:29:10.402251 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:29:10.402598 7244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 23:29:10.405532 7244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.406318 7184 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.409334 7379 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.410683 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.411077 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:10.411350 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.411705 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:10.415293 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.415660 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:10.416321 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.416724 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:10.417097 7422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36b55db1ddc34f209ac28fe8639a9a08" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:10.417428 7422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08 [sys.catalog]: This master's current role is: LEADER
W20260620 23:29:10.478000 7354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:10.478364 7354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:10.478528 7354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:10.486070 7354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:10.486219 7354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:10.486394 7354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:10.496676 7354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:36653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=43141
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:10.498581 7354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:10.499945 7354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:10.513888 7428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:10.513859 7429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:10.515630 7431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:10.517087 7354 server_base.cc:1034] running on GCE node
I20260620 23:29:10.517799 7354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:10.518850 7354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:10.520084 7354 hybrid_clock.cc:648] HybridClock initialized: now 1781998150520007 us; error 76 us; skew 500 ppm
I20260620 23:29:10.520421 7354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:10 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998150, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:10.524196 7354 init.cc:377] Logged in from keytab as oryx/127.0.96.193@KRBTEST.COM (short username oryx)
I20260620 23:29:10.526001 7354 webserver.cc:460] Webserver started at http://127.0.96.193:43141/ using document root <none> and password file <none>
I20260620 23:29:10.526804 7354 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:10.526940 7354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:10.533656 7354 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:29:10.537729 7438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:10.539840 7354 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:29:10.540135 7354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "1d9735cd69864afba13d4b00015f484c"
format_stamp: "Formatted at 2026-06-20 23:29:06 on dist-test-slave-v34r"
server_key: "e5804d8c192c33c8e884b965352bc7f5"
server_key_iv: "81c09a5d65b0731afab42b98c6b27fb2"
server_key_version: "encryptionkey@0"
I20260620 23:29:10.541625 7354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:10.559839 7354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:10.565541 7354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:10.565899 7354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:10.567186 7354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:10.569114 7354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:10.569221 7354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:10.569340 7354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:10.569427 7354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:10.613730 7354 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:36653
I20260620 23:29:10.613812 7550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:36653 every 8 connection(s)
I20260620 23:29:10.615906 7354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:10.619009 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 7354
I20260620 23:29:10.619392 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6760
Jun 20 23:29:10 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.194@KRBTEST.COM
Jun 20 23:29:10 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998150, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
Jun 20 23:29:10 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998150, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
Jun 20 23:29:10 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998150, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.252@KRBTEST.COM
I20260620 23:29:10.674614 7557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.620535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.96.194:43425)
0620 23:29:10.633046 (+ 12511us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:10.633062 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:10.645220 (+ 12158us) server_negotiation.cc:407] Connection header received
0620 23:29:10.646722 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:10.646737 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:10.647141 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:10.647294 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:10.652081 (+ 4787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.653906 (+ 1825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.657051 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.657904 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.657997 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.665297 (+ 7300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:10.665320 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:10.665352 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:10.665427 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:10.670366 (+ 4939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.671225 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.671235 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.671239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.671411 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.671855 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.671861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.671868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.672324 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:10.672627 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:10.673921 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0620 23:29:10.674133 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12295,"thread_start_us":12110,"threads_started":1}
I20260620 23:29:10.676752 6883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:10.680967 7420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.639341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:40911 (local address 127.0.96.252:34727)
0620 23:29:10.639582 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:10.639674 (+ 92us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:10.651030 (+ 11356us) server_negotiation.cc:407] Connection header received
0620 23:29:10.653009 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:10.653018 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:10.653149 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:10.653249 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:10.659667 (+ 6418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.660761 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.663583 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.664257 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.664398 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.672061 (+ 7663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:10.672068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:10.672072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:10.672115 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:10.675997 (+ 3882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.677167 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.677174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.677178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.677379 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.679316 (+ 1937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.679321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.679324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.679711 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:10.679846 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:10.680384 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 23:29:10.680610 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:29:10.682318 7558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.621462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:51523)
0620 23:29:10.639866 (+ 18404us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:10.639893 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:10.644350 (+ 4457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:10.646650 (+ 2300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:10.646675 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:10.648477 (+ 1802us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:10.650115 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.650162 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.654878 (+ 4716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.654885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.656499 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.656540 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.662977 (+ 6437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.662984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.663183 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.665243 (+ 2060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:10.665282 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:10.669011 (+ 3729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:10.676388 (+ 7377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:10.676394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:10.676398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:10.677170 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:10.677874 (+ 704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:10.677882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:10.677888 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:10.679864 (+ 1976us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x60c0000349f8. stack: 00007f415eb2566e 00007f415eb25430 00007f415dbac5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f415eb1b0fd 00007f415eb1ad1e 00007f4165632664 00007f4165534ed8 00007f4165533524 00007f416552e575 00007f416552933a 00007f41655b0dca 00007f41655aef68 00007f41655e282f
0620 23:29:10.680012 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:10.681249 (+ 1237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:10.681254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:10.681401 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 23:29:10.681558 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:29:10.681840 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18208,"spinlock_wait_cycles":2424576,"thread_start_us":15918,"threads_started":1}
I20260620 23:29:10.682334 7561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.639905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.96.193:40911)
0620 23:29:10.650869 (+ 10964us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:10.650900 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:10.652766 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:10.653542 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:10.653549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:10.654392 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:10.659455 (+ 5063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.659466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.660958 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.660962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.662677 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.662701 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.664744 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.664748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.664889 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.665633 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:10.665663 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:10.671811 (+ 6148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:10.676225 (+ 4414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:10.676233 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:10.676238 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:10.676959 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:10.677528 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:10.677533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:10.677536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:10.679154 (+ 1618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:10.679952 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:10.679957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:10.680107 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 23:29:10.680306 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 23:29:10.680876 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10734,"thread_start_us":21217,"threads_started":1}
I20260620 23:29:10.682695 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.621475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:51523 (local address 127.0.96.254:45755)
0620 23:29:10.643644 (+ 22169us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:10.643653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:10.643690 (+ 37us) server_negotiation.cc:407] Connection header received
0620 23:29:10.645381 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:10.645391 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:10.645531 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:10.645691 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:10.650421 (+ 4730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.651838 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.656757 (+ 4919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.657545 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.657696 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.669557 (+ 11861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:10.669568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:10.669576 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:10.669632 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:10.674847 (+ 5215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.677423 (+ 2576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.677433 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.677440 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.677691 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.680174 (+ 2483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.680184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.680190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.680766 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:10.680959 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:10.681746 (+ 787us) server_negotiation.cc:299] Negotiation successful
0620 23:29:10.682126 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21999,"thread_start_us":21805,"threads_started":1}
I20260620 23:29:10.683799 7555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.619961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:37961 (local address 127.0.96.253:34031)
0620 23:29:10.632455 (+ 12494us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:10.632466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:10.639095 (+ 6629us) server_negotiation.cc:407] Connection header received
0620 23:29:10.645358 (+ 6263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:10.645369 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:10.645541 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:10.645664 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:10.656879 (+ 11215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.658053 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.659400 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.660099 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.660231 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.668033 (+ 7802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:10.668040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:10.668044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:10.668087 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:10.672429 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.674533 (+ 2104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.674540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.674547 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.676421 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:10.677114 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:10.677121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:10.677177 (+ 56us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:10.677649 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:10.677785 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:10.681105 (+ 3320us) server_negotiation.cc:299] Negotiation successful
0620 23:29:10.681412 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12230,"thread_start_us":13097,"threads_started":1}
I20260620 23:29:10.684244 7552 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45755
I20260620 23:29:10.684540 7552 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:10.685307 7560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:10.620486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.96.193:37961)
0620 23:29:10.638210 (+ 17724us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:10.638390 (+ 180us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:10.645104 (+ 6714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:10.651228 (+ 6124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:10.651237 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:10.651986 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:10.652418 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.652436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.658233 (+ 5797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.658237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.659218 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:10.659229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:10.660622 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:10.660630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:10.661087 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:10.662547 (+ 1460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:10.662592 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:10.667432 (+ 4840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:10.672693 (+ 5261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:10.672707 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:10.672735 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:10.673508 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:10.676586 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:10.676591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:10.676596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:10.676925 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:10.677896 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:10.677906 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:10.679456 (+ 1550us) client_negotiation.cc:769] Sending connection context
0620 23:29:10.684113 (+ 4657us) client_negotiation.cc:240] Negotiation successful
0620 23:29:10.684595 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17129,"spinlock_wait_cycles":1805184,"thread_start_us":19552,"threads_started":1}
I20260620 23:29:10.685660 7552 heartbeater.cc:511] Master 127.0.96.254:45755 requested a full tablet report, sending...
I20260620 23:29:10.686607 7551 heartbeater.cc:347] Connected to a master server at 127.0.96.252:34727
I20260620 23:29:10.686774 7551 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:10.686882 7553 heartbeater.cc:347] Connected to a master server at 127.0.96.253:34031
I20260620 23:29:10.687100 7553 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:10.687258 7551 heartbeater.cc:511] Master 127.0.96.252:34727 requested a full tablet report, sending...
I20260620 23:29:10.687839 7553 heartbeater.cc:511] Master 127.0.96.253:34031 requested a full tablet report, sending...
I20260620 23:29:10.688447 6565 ts_manager.cc:194] Re-registered known tserver with Master: 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653)
I20260620 23:29:10.689059 7369 ts_manager.cc:194] Registered new tserver with Master: 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653)
I20260620 23:29:10.689694 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:51523
I20260620 23:29:10.690868 7174 ts_manager.cc:194] Registered new tserver with Master: 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653)
I20260620 23:29:10.776315 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6760
I20260620 23:29:10.795253 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:43425
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:45755
--webserver_port=44287
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:11.228431 7563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:11.228930 7563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:11.229143 7563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:11.237756 7563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:11.237984 7563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:11.238197 7563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:11.250597 7563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:43425
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=44287
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:11.252492 7563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:11.254257 7563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:11.271786 7568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:11.271787 7571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:11.271842 7569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:11.273130 7563 server_base.cc:1034] running on GCE node
I20260620 23:29:11.274025 7563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:11.275369 7563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:11.276638 7563 hybrid_clock.cc:648] HybridClock initialized: now 1781998151276537 us; error 85 us; skew 500 ppm
I20260620 23:29:11.277166 7563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:11 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:11.287302 7563 init.cc:377] Logged in from keytab as oryx/127.0.96.194@KRBTEST.COM (short username oryx)
I20260620 23:29:11.289928 7563 webserver.cc:460] Webserver started at http://127.0.96.194:44287/ using document root <none> and password file <none>
I20260620 23:29:11.290951 7563 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:11.291174 7563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:11.298789 7563 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:29:11.303406 7578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:11.305689 7563 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:29:11.305949 7563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "d39b41dad4494e27a2076c4c322db8d5"
format_stamp: "Formatted at 2026-06-20 23:29:07 on dist-test-slave-v34r"
server_key: "f2d091a25ffee5f4fbd0bdbf7975b968"
server_key_iv: "52070552ad672ebb26c91fce2b295653"
server_key_version: "encryptionkey@0"
I20260620 23:29:11.307188 7563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:11.323184 7563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:11.329209 7563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:11.329556 7563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:11.330955 7563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:11.332765 7563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:11.332911 7563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:11.333087 7563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:11.333189 7563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:11.379724 7563 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:43425
I20260620 23:29:11.379799 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:43425 every 8 connection(s)
I20260620 23:29:11.381685 7563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:11.383479 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 7563
I20260620 23:29:11.383774 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6894
Jun 20 23:29:11 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.195@KRBTEST.COM
Jun 20 23:29:11 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
Jun 20 23:29:11 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
Jun 20 23:29:11 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.252@KRBTEST.COM
I20260620 23:29:11.442893 7697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.385493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:40149 (local address 127.0.96.254:45755)
0620 23:29:11.412078 (+ 26585us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:11.412089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:11.415039 (+ 2950us) server_negotiation.cc:407] Connection header received
0620 23:29:11.417696 (+ 2657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:11.417703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:11.417804 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:11.417902 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:11.421165 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.422422 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.426055 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.426716 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.426824 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.433114 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:11.433123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:11.433129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:11.433236 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:11.437960 (+ 4724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.439220 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.439226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.439229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.439427 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.440104 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.440111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.440116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.440590 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:11.440731 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:11.442158 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0620 23:29:11.442522 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26363,"thread_start_us":26254,"threads_started":1}
I20260620 23:29:11.443715 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.414467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:37819 (local address 127.0.96.252:34727)
0620 23:29:11.416039 (+ 1572us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:11.416046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:11.416694 (+ 648us) server_negotiation.cc:407] Connection header received
0620 23:29:11.417722 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:11.417729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:11.417849 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:11.417972 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:11.421197 (+ 3225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.422355 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.426042 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.426830 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.426995 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.434165 (+ 7170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:11.434174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:11.434179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:11.434238 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:11.439013 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.440062 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.440068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.440071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.440240 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.440792 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.440799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.440804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.441274 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:11.441424 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:11.443012 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0620 23:29:11.443296 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1440,"thread_start_us":1308,"threads_started":1}
I20260620 23:29:11.444550 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.386953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:44053 (local address 127.0.96.253:34031)
0620 23:29:11.393545 (+ 6592us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:11.393555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:11.414445 (+ 20890us) server_negotiation.cc:407] Connection header received
0620 23:29:11.417577 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:11.417583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:11.417697 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:11.417798 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:11.421165 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.422416 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.426057 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.426716 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.426823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.433285 (+ 6462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:11.433293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:11.433301 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:11.433347 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:11.437560 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.439390 (+ 1830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.439395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.439398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.439590 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.440486 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.440491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.440494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.440871 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:11.440964 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:11.443720 (+ 2756us) server_negotiation.cc:299] Negotiation successful
0620 23:29:11.444201 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6400,"thread_start_us":6241,"threads_started":1}
I20260620 23:29:11.444684 7702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.415234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.96.194:37819)
0620 23:29:11.416446 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:11.416474 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:11.417151 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:11.418384 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:11.418402 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:11.419655 (+ 1253us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:11.420928 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:11.420945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.422590 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.422597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:11.425781 (+ 3184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:11.425804 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.427366 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.427376 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:11.427807 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.429053 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:11.429081 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:11.433853 (+ 4772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:11.439268 (+ 5415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:11.439275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:11.439280 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:11.439885 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:11.440428 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:11.440437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:11.440442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:11.440647 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:11.441494 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:11.441500 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:11.441692 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 23:29:11.442047 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 23:29:11.442840 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":495,"spinlock_wait_cycles":68096,"thread_start_us":818,"threads_started":1}
I20260620 23:29:11.445637 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.386515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:40149)
0620 23:29:11.413358 (+ 26843us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:11.414288 (+ 930us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:11.417119 (+ 2831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:11.418384 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:11.418400 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:11.419597 (+ 1197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:11.420874 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:11.420910 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.422641 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.422647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:11.425838 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:11.425859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.427206 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.427215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:11.427795 (+ 580us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.428853 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:11.428901 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:11.432744 (+ 3843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:11.438540 (+ 5796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:11.438547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:11.438551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:11.439065 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:11.439723 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:11.439726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:11.439729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:11.439985 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:11.441093 (+ 1108us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:11.441102 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:11.441690 (+ 588us) client_negotiation.cc:769] Sending connection context
0620 23:29:11.442045 (+ 355us) client_negotiation.cc:240] Negotiation successful
0620 23:29:11.444085 (+ 2040us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26382,"mutex_wait_us":2109,"spinlock_wait_cycles":51712,"thread_start_us":26149,"threads_started":1}
I20260620 23:29:11.445665 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.386518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.96.194:44053)
0620 23:29:11.414268 (+ 27750us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:11.414332 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:11.417119 (+ 2787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:11.418384 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:11.418400 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:11.419598 (+ 1198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:11.420874 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:11.420910 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.422641 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.422647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:11.425838 (+ 3191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:11.425859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.427206 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.427215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:11.427787 (+ 572us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.428871 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:11.428901 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:11.432960 (+ 4059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:11.437833 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:11.437846 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:11.437868 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:11.439256 (+ 1388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:11.440135 (+ 879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:11.440139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:11.440142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:11.440293 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:11.441102 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:11.441106 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:11.441683 (+ 577us) client_negotiation.cc:769] Sending connection context
0620 23:29:11.444808 (+ 3125us) client_negotiation.cc:240] Negotiation successful
0620 23:29:11.445168 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27557,"mutex_wait_us":1252,"spinlock_wait_cycles":134272,"thread_start_us":25377,"threads_started":1}
I20260620 23:29:11.446070 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:11.384914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.96.195:34527)
0620 23:29:11.413973 (+ 29059us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:11.413987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:11.414761 (+ 774us) server_negotiation.cc:407] Connection header received
0620 23:29:11.414891 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:11.414900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:11.415286 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:11.415508 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:11.418583 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.420422 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.422676 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:11.423771 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:11.423929 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:11.434682 (+ 10753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:11.434714 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:11.434741 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:11.434820 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:11.440971 (+ 6151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.442159 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.442170 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.442173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.442387 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:11.442941 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:11.442951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:11.442958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:11.443534 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:11.443892 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:11.445256 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0620 23:29:11.445612 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28822,"thread_start_us":28654,"threads_started":1}
I20260620 23:29:11.446589 7691 heartbeater.cc:347] Connected to a master server at 127.0.96.252:34727
I20260620 23:29:11.447000 7691 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:11.447362 7693 heartbeater.cc:347] Connected to a master server at 127.0.96.253:34031
I20260620 23:29:11.447573 7693 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:11.448185 7017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:11.448201 7691 heartbeater.cc:511] Master 127.0.96.252:34727 requested a full tablet report, sending...
I20260620 23:29:11.448223 7693 heartbeater.cc:511] Master 127.0.96.253:34031 requested a full tablet report, sending...
I20260620 23:29:11.448881 7692 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45755
I20260620 23:29:11.449126 7692 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:11.449677 7692 heartbeater.cc:511] Master 127.0.96.254:45755 requested a full tablet report, sending...
I20260620 23:29:11.450577 7369 ts_manager.cc:194] Registered new tserver with Master: d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425)
I20260620 23:29:11.451047 7174 ts_manager.cc:194] Registered new tserver with Master: d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425)
I20260620 23:29:11.451489 6565 ts_manager.cc:194] Re-registered known tserver with Master: d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425)
I20260620 23:29:11.452420 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:40149
I20260620 23:29:11.542563 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6894
I20260620 23:29:11.561504 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:34527
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:45755
--webserver_port=46117
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:11.692330 7552 heartbeater.cc:503] Master 127.0.96.254:45755 was elected leader, sending a full tablet report...
W20260620 23:29:11.920432 7704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:11.920734 7704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:11.920892 7704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:11.928325 7704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:11.928465 7704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:11.928654 7704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:11.939121 7704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:34527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=46117
--tserver_master_addrs=127.0.96.254:45755,127.0.96.253:34031,127.0.96.252:34727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:11.940510 7704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:11.941891 7704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:11.957950 7709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:11.958050 7710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:11.958657 7712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:11.959638 7704 server_base.cc:1034] running on GCE node
I20260620 23:29:11.960371 7704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:11.961791 7704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:11.963119 7704 hybrid_clock.cc:648] HybridClock initialized: now 1781998151963030 us; error 84 us; skew 500 ppm
I20260620 23:29:11.963521 7704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:11 dist-test-slave-v34r krb5kdc[6520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:11.967415 7704 init.cc:377] Logged in from keytab as oryx/127.0.96.195@KRBTEST.COM (short username oryx)
I20260620 23:29:11.969305 7704 webserver.cc:460] Webserver started at http://127.0.96.195:46117/ using document root <none> and password file <none>
I20260620 23:29:11.970103 7704 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:11.970284 7704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:11.976609 7704 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 23:29:11.980317 7719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:11.982077 7704 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260620 23:29:11.982311 7704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "4ddded06fcea475ea96243bb51197b85"
format_stamp: "Formatted at 2026-06-20 23:29:08 on dist-test-slave-v34r"
server_key: "b7b919839f8a72bf9fac5a1575b13205"
server_key_iv: "b0eecb8f70c3bea45c0bc987498a5094"
server_key_version: "encryptionkey@0"
I20260620 23:29:11.983413 7704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:12.000620 7704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:12.004971 7704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:12.005336 7704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:12.006450 7704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:12.007968 7704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:12.008097 7704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:12.008244 7704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:12.008335 7704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:12.052603 7704 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:34527
I20260620 23:29:12.052651 7831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:34527 every 8 connection(s)
I20260620 23:29:12.054178 7704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:12.058892 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 7704
Jun 20 23:29:12 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.253@KRBTEST.COM
Jun 20 23:29:12 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
Jun 20 23:29:12 dist-test-slave-v34r krb5kdc[6520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998151, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.252@KRBTEST.COM
I20260620 23:29:12.094564 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.062696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:57177 (local address 127.0.96.254:45755)
0620 23:29:12.069437 (+ 6741us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.069444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.069463 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:29:12.069518 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.069523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.069602 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.071816 (+ 2214us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.071972 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.073026 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.076084 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.076736 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.077752 (+ 1016us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.084353 (+ 6601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.084360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.084364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.084414 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.088364 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.089870 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.089880 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.089889 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.090102 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.091611 (+ 1509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.091618 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.091624 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.092134 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.092322 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.093760 (+ 1438us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.094084 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6588,"thread_start_us":6481,"threads_started":1}
I20260620 23:29:12.096539 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.062250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:57177)
0620 23:29:12.066520 (+ 4270us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.066543 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.068381 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.069785 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.069789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.070425 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:12.070720 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.070728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.073183 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.073188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.074738 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.074760 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.077080 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.077091 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.077613 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.079367 (+ 1754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:12.079419 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:12.084011 (+ 4592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:12.088566 (+ 4555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:12.088583 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:12.088612 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:12.089558 (+ 946us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:12.091070 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:12.091078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:12.091086 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:12.091448 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:12.092464 (+ 1016us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:12.092477 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:12.093174 (+ 697us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.093622 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.094409 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4125,"mutex_wait_us":683,"spinlock_wait_cycles":7040,"thread_start_us":3978,"threads_started":1}
I20260620 23:29:12.098773 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.059619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.96.195:37191)
0620 23:29:12.060676 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.060698 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.067543 (+ 6845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.074292 (+ 6749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.074297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.074880 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:12.075200 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.075210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.077027 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.077036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.079104 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.079135 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.080965 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.080975 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.081264 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.082045 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:12.082077 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:12.086821 (+ 4744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:12.092217 (+ 5396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:12.092225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:12.092229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:12.093472 (+ 1243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:12.094122 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:12.094127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:12.094131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:12.094314 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:12.097339 (+ 3025us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:12.097349 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:12.097630 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.097831 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.098255 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":5,"spinlock_wait_cycles":84736,"thread_start_us":1453,"threads_started":1}
I20260620 23:29:12.099597 7833 heartbeater.cc:347] Connected to a master server at 127.0.96.254:45755
I20260620 23:29:12.099817 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.058000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:54683 (local address 127.0.96.253:34031)
0620 23:29:12.058496 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.058500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.060562 (+ 2062us) server_negotiation.cc:407] Connection header received
0620 23:29:12.068040 (+ 7478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.068045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.068115 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.068240 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.070889 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.071594 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.074914 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.075709 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.075837 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.084768 (+ 8931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.084777 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.084783 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.084836 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.090844 (+ 6008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.096962 (+ 6118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.096967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.096970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.097210 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.097740 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.097745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.097748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.098125 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.098395 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.099034 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.099300 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":287,"threads_started":1}
I20260620 23:29:12.099961 7833 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:12.100371 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.060796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:37191 (local address 127.0.96.252:34727)
0620 23:29:12.073693 (+ 12897us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.073704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.073745 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:29:12.073847 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.073856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.074023 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.074255 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.075568 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.076778 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.079440 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.080317 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.080433 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.087152 (+ 6719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.087161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.087165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.087219 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.091964 (+ 4745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.093660 (+ 1696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.093666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.093670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.093836 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.094450 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.094454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.094457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.094870 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.095065 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.099389 (+ 4324us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.099788 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12636,"thread_start_us":12482,"threads_started":1}
I20260620 23:29:12.101600 7832 heartbeater.cc:347] Connected to a master server at 127.0.96.252:34727
I20260620 23:29:12.101789 7832 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:12.102674 7832 heartbeater.cc:511] Master 127.0.96.252:34727 requested a full tablet report, sending...
I20260620 23:29:12.103875 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.058972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.96.195:54683)
0620 23:29:12.060322 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.060404 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.067543 (+ 7139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.068550 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.068559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.069496 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:12.070434 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.070458 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.071765 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.071774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.073892 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.073906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.077125 (+ 3219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.077131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.077611 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.079385 (+ 1774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:12.079415 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:12.084498 (+ 5083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:12.091609 (+ 7111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:12.091617 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:12.091623 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:12.092261 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:12.097379 (+ 5118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:12.097385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:12.097389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:12.097567 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:12.098397 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:12.098404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:12.098603 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.102369 (+ 3766us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.103025 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":1248,"spinlock_wait_cycles":1152,"thread_start_us":736,"threads_started":1}
I20260620 23:29:12.104353 7833 heartbeater.cc:511] Master 127.0.96.254:45755 requested a full tablet report, sending...
I20260620 23:29:12.105393 7369 ts_manager.cc:194] Registered new tserver with Master: 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527)
I20260620 23:29:12.106081 6565 ts_manager.cc:194] Re-registered known tserver with Master: 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527)
I20260620 23:29:12.106314 7834 heartbeater.cc:347] Connected to a master server at 127.0.96.253:34031
I20260620 23:29:12.106516 7834 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:12.107148 7834 heartbeater.cc:511] Master 127.0.96.253:34031 requested a full tablet report, sending...
I20260620 23:29:12.107350 6565 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:57177
I20260620 23:29:12.108709 7174 ts_manager.cc:194] Registered new tserver with Master: 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527)
I20260620 23:29:12.128455 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.084022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.0.96.252:34727)
0620 23:29:12.085069 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.085076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.096226 (+ 11150us) server_negotiation.cc:407] Connection header received
0620 23:29:12.096466 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.096474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.096596 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.096757 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.109289 (+ 12532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.110960 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.114530 (+ 3570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.115798 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.115987 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.120605 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.120611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.120615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.120653 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.125092 (+ 4439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.126118 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.126124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.126128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.126287 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.126713 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.126720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.126725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.127116 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.127363 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.127820 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.128029 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":1520,"threads_started":1}
I20260620 23:29:12.130263 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.078105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.0.96.254:45755)
0620 23:29:12.078591 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.078597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.081904 (+ 3307us) server_negotiation.cc:407] Connection header received
0620 23:29:12.082226 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.082232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.082315 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.082398 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.109187 (+ 26789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.110348 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.116368 (+ 6020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.117075 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.117228 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.121094 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.121102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.121108 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.121157 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.124409 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.125749 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.125758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.125761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.125930 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.126910 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.126916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.126919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.127354 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.127606 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.129384 (+ 1778us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.129839 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":643,"threads_started":1}
I20260620 23:29:12.134424 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.089206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.0.96.253:34031)
0620 23:29:12.100518 (+ 11312us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.100524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.113775 (+ 13251us) server_negotiation.cc:407] Connection header received
0620 23:29:12.113868 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.113873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.114007 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.114115 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.117534 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.118648 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.121100 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.121752 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.121935 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.125860 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.125867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.125872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.125916 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.129445 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.131436 (+ 1991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.131441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.131444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.131663 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.132116 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.132121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.132124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.132510 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.132803 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.133764 (+ 961us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.134034 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10760,"thread_start_us":23487,"threads_started":1}
I20260620 23:29:12.135267 6565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:29:12.146104 6565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:12.179056 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.170534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:43425 (local address 127.0.0.1:40782)
0620 23:29:12.171257 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.171282 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.171429 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.172118 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.172123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.172147 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:12.172521 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.172533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.174171 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.174177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.175872 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.175884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.177914 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.177920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.178147 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.178196 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.178481 (+ 285us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.178650 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":1973,"threads_started":1}
I20260620 23:29:12.179714 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.170990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.0.96.194:43425)
0620 23:29:12.171574 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.171584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.171617 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:29:12.171701 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.171708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.171926 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.172140 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:12.172751 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.174036 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.176031 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.177338 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.177492 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.177770 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.179276 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.179422 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":885,"threads_started":1}
I20260620 23:29:12.185227 7626 tablet_service.cc:1467] Processing CreateTablet for tablet c79e1fe17546435688c161b35f92a6ca (DEFAULT_TABLE table=test-table [id=b1e378e29e4e45c7ad91552bc8749691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:12.187054 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c79e1fe17546435688c161b35f92a6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:12.187635 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.171006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:34527 (local address 127.0.0.1:53298)
0620 23:29:12.178661 (+ 7655us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.178680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.178816 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.180517 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.180525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.180557 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:12.181395 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.181406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.183062 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.183070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.184711 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.184723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.186815 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.186821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.186982 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.187018 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.187137 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.187274 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7508,"thread_start_us":7218,"threads_started":1}
I20260620 23:29:12.187722 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.171546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.0.96.195:34527)
0620 23:29:12.178034 (+ 6488us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.178045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.178814 (+ 769us) server_negotiation.cc:407] Connection header received
0620 23:29:12.178994 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.179004 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.180196 (+ 1192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.180543 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:12.181623 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.182932 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.185089 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.186613 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.186760 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.186996 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.187188 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.187415 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6289,"thread_start_us":7248,"threads_started":1}
I20260620 23:29:12.190644 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.179409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36653 (local address 127.0.0.1:60678)
0620 23:29:12.179633 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.179653 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.180057 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.182369 (+ 2312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.182373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.182399 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:12.182568 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.182577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.184398 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.184403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.186234 (+ 1831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.186248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.189743 (+ 3495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.189750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.189925 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.189954 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.190088 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.190313 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:29:12.190676 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.178020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.96.193:36653)
0620 23:29:12.181637 (+ 3617us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.181649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.181684 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:29:12.181804 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.181813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.182121 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.182358 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:12.182751 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.184217 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.186423 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.189150 (+ 2727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.189356 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.189605 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.190139 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.190349 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3384,"thread_start_us":11020,"threads_started":1}
I20260620 23:29:12.192812 7767 tablet_service.cc:1467] Processing CreateTablet for tablet c79e1fe17546435688c161b35f92a6ca (DEFAULT_TABLE table=test-table [id=b1e378e29e4e45c7ad91552bc8749691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:12.195128 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c79e1fe17546435688c161b35f92a6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:12.194996 7486 tablet_service.cc:1467] Processing CreateTablet for tablet c79e1fe17546435688c161b35f92a6ca (DEFAULT_TABLE table=test-table [id=b1e378e29e4e45c7ad91552bc8749691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:12.196796 7486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c79e1fe17546435688c161b35f92a6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:12.207756 7868 tablet_bootstrap.cc:492] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Bootstrap starting.
I20260620 23:29:12.213773 7869 tablet_bootstrap.cc:492] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Bootstrap starting.
I20260620 23:29:12.214213 7868 tablet_bootstrap.cc:654] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:12.216562 7868 log.cc:824] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:12.217552 7869 tablet_bootstrap.cc:654] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:12.218232 7870 tablet_bootstrap.cc:492] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Bootstrap starting.
I20260620 23:29:12.218880 7869 log.cc:824] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:12.223608 7870 tablet_bootstrap.cc:654] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:12.225297 7870 log.cc:824] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:12.227070 7868 tablet_bootstrap.cc:492] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: No bootstrap required, opened a new log
I20260620 23:29:12.227445 7868 ts_tablet_manager.cc:1397] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20260620 23:29:12.227741 7869 tablet_bootstrap.cc:492] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: No bootstrap required, opened a new log
I20260620 23:29:12.228099 7869 ts_tablet_manager.cc:1397] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260620 23:29:12.228762 7870 tablet_bootstrap.cc:492] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: No bootstrap required, opened a new log
I20260620 23:29:12.229120 7870 ts_tablet_manager.cc:1397] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20260620 23:29:12.237385 7868 raft_consensus.cc:348] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.237957 7868 raft_consensus.cc:374] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:12.238065 7868 raft_consensus.cc:729] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39b41dad4494e27a2076c4c322db8d5, State: Initialized, Role: FOLLOWER
I20260620 23:29:12.238845 7868 consensus_queue.cc:260] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.241196 7868 ts_tablet_manager.cc:1428] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260620 23:29:12.241043 7870 raft_consensus.cc:348] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.241539 7870 raft_consensus.cc:374] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:12.241593 7692 heartbeater.cc:503] Master 127.0.96.254:45755 was elected leader, sending a full tablet report...
I20260620 23:29:12.241641 7870 raft_consensus.cc:729] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9735cd69864afba13d4b00015f484c, State: Initialized, Role: FOLLOWER
I20260620 23:29:12.242393 7870 consensus_queue.cc:260] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.241048 7869 raft_consensus.cc:348] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.243006 7869 raft_consensus.cc:374] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:12.243135 7869 raft_consensus.cc:729] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddded06fcea475ea96243bb51197b85, State: Initialized, Role: FOLLOWER
I20260620 23:29:12.243929 7869 consensus_queue.cc:260] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.248663 7869 ts_tablet_manager.cc:1428] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Time spent starting tablet: real 0.020s user 0.017s sys 0.003s
I20260620 23:29:12.249086 7833 heartbeater.cc:503] Master 127.0.96.254:45755 was elected leader, sending a full tablet report...
I20260620 23:29:12.252409 7870 ts_tablet_manager.cc:1428] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Time spent starting tablet: real 0.023s user 0.015s sys 0.005s
W20260620 23:29:12.308203 7835 tablet.cc:2369] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:12.367264 7876 raft_consensus.cc:482] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:12.367707 7876 raft_consensus.cc:504] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.370040 7876 leader_election.cc:288] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425), 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653)
I20260620 23:29:12.379335 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.370686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:50845 (local address 127.0.96.194:43425)
0620 23:29:12.370945 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.370953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.370980 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:29:12.371072 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.371081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.371180 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.371297 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:12.371769 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.373120 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.376362 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.377882 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.378094 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.378178 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.378855 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.379031 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260620 23:29:12.379415 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.370372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:43425 (local address 127.0.96.195:50845)
0620 23:29:12.370686 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.370710 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.370903 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.371341 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.371349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.371375 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:12.371569 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.371578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.373329 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.373336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.376091 (+ 2755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.376117 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.378282 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.378290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.378581 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.378628 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.378762 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.378941 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":189}
I20260620 23:29:12.380671 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79e1fe17546435688c161b35f92a6ca" candidate_uuid: "4ddded06fcea475ea96243bb51197b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39b41dad4494e27a2076c4c322db8d5" is_pre_election: true
I20260620 23:29:12.381618 7646 raft_consensus.cc:2454] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ddded06fcea475ea96243bb51197b85 in term 0.
I20260620 23:29:12.382584 7721 leader_election.cc:302] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ddded06fcea475ea96243bb51197b85, d39b41dad4494e27a2076c4c322db8d5; no voters:
I20260620 23:29:12.382717 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.370851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36653 (local address 127.0.96.195:34223)
0620 23:29:12.371114 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:12.371136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:12.371297 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:12.371695 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:12.371699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:12.371724 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:12.371864 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.371870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.373342 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.373346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.376091 (+ 2745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:12.376117 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.378358 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.378365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:12.382045 (+ 3680us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.382123 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 23:29:12.382265 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 23:29:12.382459 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":151}
I20260620 23:29:12.383396 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.371095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:34223 (local address 127.0.96.193:36653)
0620 23:29:12.371383 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.371391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.371413 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:29:12.371496 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.371500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.371561 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.371645 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:12.372002 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.373183 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.376355 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.377942 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.378101 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.378178 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.382907 (+ 4729us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.383122 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 23:29:12.383351 7876 raft_consensus.cc:2781] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:12.383978 7876 raft_consensus.cc:482] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:12.384152 7876 raft_consensus.cc:3037] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:12.384532 7506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79e1fe17546435688c161b35f92a6ca" candidate_uuid: "4ddded06fcea475ea96243bb51197b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9735cd69864afba13d4b00015f484c" is_pre_election: true
I20260620 23:29:12.385226 7506 raft_consensus.cc:2454] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ddded06fcea475ea96243bb51197b85 in term 0.
I20260620 23:29:12.387468 7876 raft_consensus.cc:504] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.388235 7876 leader_election.cc:288] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [CANDIDATE]: Term 1 election: Requested vote from peers d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425), 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653)
I20260620 23:29:12.389643 7646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79e1fe17546435688c161b35f92a6ca" candidate_uuid: "4ddded06fcea475ea96243bb51197b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d39b41dad4494e27a2076c4c322db8d5"
I20260620 23:29:12.389935 7646 raft_consensus.cc:3037] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:12.390184 7506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c79e1fe17546435688c161b35f92a6ca" candidate_uuid: "4ddded06fcea475ea96243bb51197b85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9735cd69864afba13d4b00015f484c"
I20260620 23:29:12.390444 7506 raft_consensus.cc:3037] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Advancing to term 1
W20260620 23:29:12.392642 7694 tablet.cc:2369] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:12.394089 7646 raft_consensus.cc:2454] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ddded06fcea475ea96243bb51197b85 in term 1.
I20260620 23:29:12.394660 7506 raft_consensus.cc:2454] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ddded06fcea475ea96243bb51197b85 in term 1.
I20260620 23:29:12.394766 7721 leader_election.cc:302] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ddded06fcea475ea96243bb51197b85, d39b41dad4494e27a2076c4c322db8d5; no voters:
I20260620 23:29:12.395218 7876 raft_consensus.cc:2781] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:12.395848 7876 raft_consensus.cc:686] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 1 LEADER]: Becoming Leader. State: Replica: 4ddded06fcea475ea96243bb51197b85, State: Running, Role: LEADER
I20260620 23:29:12.396484 7876 consensus_queue.cc:237] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } }
I20260620 23:29:12.402359 6564 catalog_manager.cc:5495] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ddded06fcea475ea96243bb51197b85 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "4ddded06fcea475ea96243bb51197b85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } health_report { overall_health: UNKNOWN } } }
W20260620 23:29:12.406773 7554 tablet.cc:2369] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:12.440341 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.434357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.0.96.195:34527)
0620 23:29:12.434611 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.434619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.434783 (+ 164us) server_negotiation.cc:407] Connection header received
0620 23:29:12.435050 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.435054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.435143 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.435328 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:12.435723 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.436471 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.437962 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.438417 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.438549 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.439123 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:12.439735 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:12.439912 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.439971 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.440051 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:29:12.452070 7646 raft_consensus.cc:1261] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Refusing update from remote peer 4ddded06fcea475ea96243bb51197b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:12.452070 7506 raft_consensus.cc:1261] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Refusing update from remote peer 4ddded06fcea475ea96243bb51197b85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:12.453531 7877 consensus_queue.cc:1035] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:12.453961 7876 consensus_queue.cc:1035] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:12.471823 7882 mvcc.cc:205] Tried to move back new op lower bound from 7299064432425533440 to 7299064432220045312. Current Snapshot: MvccSnapshot[applied={T|T < 7299064432425533440}]
I20260620 23:29:12.494640 6564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:12.495033 6564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:12.503125 6564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36b55db1ddc34f209ac28fe8639a9a08: Sending DeleteTablet for 3 replicas of tablet c79e1fe17546435688c161b35f92a6ca
I20260620 23:29:12.504748 7626 tablet_service.cc:1514] Processing DeleteTablet for tablet c79e1fe17546435688c161b35f92a6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:12 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:40782
I20260620 23:29:12.504844 7767 tablet_service.cc:1514] Processing DeleteTablet for tablet c79e1fe17546435688c161b35f92a6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:12 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:53298
I20260620 23:29:12.504889 7486 tablet_service.cc:1514] Processing DeleteTablet for tablet c79e1fe17546435688c161b35f92a6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:12 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:60678
I20260620 23:29:12.517858 7891 tablet_replica.cc:330] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: stopping tablet replica
I20260620 23:29:12.518373 7891 raft_consensus.cc:2229] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:12.520817 7890 tablet_replica.cc:330] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: stopping tablet replica
I20260620 23:29:12.520865 7889 tablet_replica.cc:330] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: stopping tablet replica
I20260620 23:29:12.520969 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 6533
I20260620 23:29:12.521539 7889 raft_consensus.cc:2229] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:12.522467 7889 raft_consensus.cc:2258] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:12.523268 7891 raft_consensus.cc:2258] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:12.523874 6595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:12.524178 7890 raft_consensus.cc:2229] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:12.524814 7890 raft_consensus.cc:2258] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:12.526923 7891 ts_tablet_manager.cc:1905] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:12.526922 7890 ts_tablet_manager.cc:1905] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:12.528406 7889 ts_tablet_manager.cc:1905] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:12.535976 7890 ts_tablet_manager.cc:1918] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:12.536216 7890 log.cc:1196] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/c79e1fe17546435688c161b35f92a6ca
I20260620 23:29:12.536883 7890 ts_tablet_manager.cc:1939] T c79e1fe17546435688c161b35f92a6ca P d39b41dad4494e27a2076c4c322db8d5: Deleting consensus metadata
I20260620 23:29:12.545183 7889 ts_tablet_manager.cc:1918] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:12.545516 7889 log.cc:1196] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/c79e1fe17546435688c161b35f92a6ca
I20260620 23:29:12.546212 7889 ts_tablet_manager.cc:1939] T c79e1fe17546435688c161b35f92a6ca P 4ddded06fcea475ea96243bb51197b85: Deleting consensus metadata
I20260620 23:29:12.550413 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.507715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.0.96.253:34031)
0620 23:29:12.507904 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.507911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.519368 (+ 11457us) server_negotiation.cc:407] Connection header received
0620 23:29:12.519987 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.519995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.520094 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.520237 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.524521 (+ 4284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.526115 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.530932 (+ 4817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.531841 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.532032 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.540673 (+ 8641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.540682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.540686 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.540744 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.544657 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.545846 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.545852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.545857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.546123 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.546715 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.546720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.546723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.547220 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.547597 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.549725 (+ 2128us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.550003 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:29:12.551156 7891 ts_tablet_manager.cc:1918] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:12.551369 7891 log.cc:1196] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/c79e1fe17546435688c161b35f92a6ca
I20260620 23:29:12.552052 7891 ts_tablet_manager.cc:1939] T c79e1fe17546435688c161b35f92a6ca P 1d9735cd69864afba13d4b00015f484c: Deleting consensus metadata
I20260620 23:29:12.566462 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.519606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.0.96.252:34727)
0620 23:29:12.519848 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.519855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.551311 (+ 31456us) server_negotiation.cc:407] Connection header received
0620 23:29:12.551578 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.551586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.551685 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.551820 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.554124 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.555187 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.556655 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.557241 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.557412 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.560391 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.560397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.560401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.560444 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.563449 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.564260 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.564268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.564274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.564421 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.564817 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.564823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.564826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.565236 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.565492 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.565845 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.566092 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:29:12.710909 6550 catalog_manager.cc:4841] TS 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653): tablet c79e1fe17546435688c161b35f92a6ca (table test-table [id=b1e378e29e4e45c7ad91552bc8749691]) successfully deleted
I20260620 23:29:12.711355 6550 catalog_manager.cc:4841] TS d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425): tablet c79e1fe17546435688c161b35f92a6ca (table test-table [id=b1e378e29e4e45c7ad91552bc8749691]) successfully deleted
I20260620 23:29:12.711664 6551 catalog_manager.cc:4841] TS 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527): tablet c79e1fe17546435688c161b35f92a6ca (table test-table [id=b1e378e29e4e45c7ad91552bc8749691]) successfully deleted
I20260620 23:29:12.712271 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 6533
I20260620 23:29:12.773279 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.750376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.96.253:34031)
0620 23:29:12.750631 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.750641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.751677 (+ 1036us) server_negotiation.cc:407] Connection header received
0620 23:29:12.751980 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.751989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.752092 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.752184 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.755475 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.756935 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.759496 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.760393 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.760514 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.765990 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.765999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.766005 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.766054 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.769838 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.770765 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.770774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.770778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.771003 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.771481 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.771485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.771487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.771820 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.772131 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.772507 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.772831 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:29:12.778275 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:12.751704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.96.252:34727)
0620 23:29:12.751881 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:12.751890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:12.762196 (+ 10306us) server_negotiation.cc:407] Connection header received
0620 23:29:12.762522 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:12.762529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:12.762617 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:12.762703 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:12.764703 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.765586 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.767415 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:12.768127 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:12.768278 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:12.771465 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:12.771474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:12.771482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:12.771540 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:12.775075 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.775824 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.775830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.775834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.775963 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:12.776325 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:12.776331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:12.776334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:12.776664 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:12.776963 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:12.777623 (+ 660us) server_negotiation.cc:299] Negotiation successful
0620 23:29:12.777933 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:29:13.485440 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:13.484458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:59035)
0620 23:29:13.484963 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:13.485179 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
W20260620 23:29:13.486002 7833 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:45755 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:13.487908 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:13.486657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:35471)
0620 23:29:13.487236 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:13.487349 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":336,"threads_started":1}
W20260620 23:29:13.488798 7692 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:45755 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:14.003937 7904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36b55db1ddc34f209ac28fe8639a9a08)
I20260620 23:29:14.004271 7905 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 36b55db1ddc34f209ac28fe8639a9a08)
I20260620 23:29:14.004359 7904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:14.004567 7905 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:14.006197 7905 leader_election.cc:288] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755), 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252:34727)
I20260620 23:29:14.006258 7904 leader_election.cc:288] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755), ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253:34031)
I20260620 23:29:14.007674 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.006413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33334)
0620 23:29:14.007144 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.007354 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":368,"threads_started":1}
I20260620 23:29:14.007997 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.006408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33320)
0620 23:29:14.007477 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.007682 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":940,"thread_start_us":1028,"threads_started":1}
W20260620 23:29:14.008137 7353 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
W20260620 23:29:14.010629 7159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:14.012763 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.009808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33340)
0620 23:29:14.011987 (+ 2179us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.012382 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1993}
W20260620 23:29:14.013031 7353 leader_election.cc:334] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:14.014024 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.013143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33352)
0620 23:29:14.013417 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.013650 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260620 23:29:14.014298 7159 leader_election.cc:334] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:14.019472 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.007899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:34727 (local address 127.0.0.1:55566)
0620 23:29:14.008344 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.008366 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:14.008614 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:14.013024 (+ 4410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:14.013033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:14.013066 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:14.013362 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:14.013374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.015029 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.015036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:14.017245 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:14.017260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.018613 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.018620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:14.018826 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:14.018848 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 23:29:14.018966 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:29:14.019126 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":2333,"threads_started":1}
I20260620 23:29:14.019608 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.008438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.0.96.252:34727)
0620 23:29:14.012491 (+ 4053us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:14.012497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:14.012593 (+ 96us) server_negotiation.cc:407] Connection header received
0620 23:29:14.012675 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:14.012681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:14.012796 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:14.013608 (+ 812us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:14.013877 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.014923 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.017486 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.018337 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.018555 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:14.018713 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:14.019009 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 23:29:14.019204 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3880,"thread_start_us":3689,"threads_started":1}
I20260620 23:29:14.019905 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.006829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:34031 (local address 127.0.0.1:48342)
0620 23:29:14.009280 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.009300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:14.009450 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:14.012780 (+ 3330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:14.012789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:14.012831 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:14.013232 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:14.013246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.014971 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.014977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:14.017242 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:14.017260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.018838 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.018845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:14.019078 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:14.019119 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:29:14.019241 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:29:14.019457 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2332,"thread_start_us":4770,"threads_started":1}
I20260620 23:29:14.020411 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.007274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.0.96.253:34031)
0620 23:29:14.010836 (+ 3562us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:14.010842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:14.010862 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:14.010933 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:14.010938 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:14.011041 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:14.011161 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:14.013499 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.014820 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.017468 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:14.018443 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:14.018603 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:14.018733 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:14.019733 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0620 23:29:14.019941 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":3422,"thread_start_us":3235,"threads_started":1}
I20260620 23:29:14.021082 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "643e504e8b2d4db28aeee4b200eb73eb" is_pre_election: true
I20260620 23:29:14.021260 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "643e504e8b2d4db28aeee4b200eb73eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" is_pre_election: true
I20260620 23:29:14.021513 7379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd4a7787ced42269d1b4f3d43cd9772 in term 1.
I20260620 23:29:14.021548 7184 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 643e504e8b2d4db28aeee4b200eb73eb in term 1.
I20260620 23:29:14.022118 7159 leader_election.cc:302] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 643e504e8b2d4db28aeee4b200eb73eb, ffd4a7787ced42269d1b4f3d43cd9772; no voters: 36b55db1ddc34f209ac28fe8639a9a08
I20260620 23:29:14.022119 7356 leader_election.cc:302] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 643e504e8b2d4db28aeee4b200eb73eb, ffd4a7787ced42269d1b4f3d43cd9772; no voters: 36b55db1ddc34f209ac28fe8639a9a08
I20260620 23:29:14.022471 7905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:29:14.022584 7905 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36b55db1ddc34f209ac28fe8639a9a08)
I20260620 23:29:14.022680 7905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:29:14.023653 7904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 23:29:14.023834 7904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 36b55db1ddc34f209ac28fe8639a9a08)
I20260620 23:29:14.023948 7904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:29:14.027750 7905 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:14.028515 7905 leader_election.cc:288] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 2 election: Requested vote from peers 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755), 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252:34727)
I20260620 23:29:14.029719 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.028875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33370)
0620 23:29:14.029451 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.029541 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467}
I20260620 23:29:14.029579 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "643e504e8b2d4db28aeee4b200eb73eb"
I20260620 23:29:14.030251 7904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:14.030932 7379 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ffd4a7787ced42269d1b4f3d43cd9772 in current term 2: Already voted for candidate 643e504e8b2d4db28aeee4b200eb73eb in this term.
I20260620 23:29:14.031244 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.030812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33382)
0620 23:29:14.031028 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.031108 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 23:29:14.031415 7159 leader_election.cc:334] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:14.031620 7904 leader_election.cc:288] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [CANDIDATE]: Term 2 election: Requested vote from peers 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755), ffd4a7787ced42269d1b4f3d43cd9772 (127.0.96.253:34031)
I20260620 23:29:14.031909 7159 leader_election.cc:302] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffd4a7787ced42269d1b4f3d43cd9772; no voters: 36b55db1ddc34f209ac28fe8639a9a08, 643e504e8b2d4db28aeee4b200eb73eb
I20260620 23:29:14.032171 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.031300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33390)
0620 23:29:14.031865 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.031970 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
I20260620 23:29:14.032136 7184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "643e504e8b2d4db28aeee4b200eb73eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ffd4a7787ced42269d1b4f3d43cd9772"
I20260620 23:29:14.032205 7905 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:29:14.032550 7184 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 643e504e8b2d4db28aeee4b200eb73eb in current term 2: Already voted for candidate ffd4a7787ced42269d1b4f3d43cd9772 in this term.
I20260620 23:29:14.033182 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.032751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33404)
0620 23:29:14.032895 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.033056 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 23:29:14.033389 7353 leader_election.cc:334] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:14.033499 7353 leader_election.cc:302] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 643e504e8b2d4db28aeee4b200eb73eb; no voters: 36b55db1ddc34f209ac28fe8639a9a08, ffd4a7787ced42269d1b4f3d43cd9772
I20260620 23:29:14.033751 7904 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 23:29:14.488003 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.486938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:35901)
0620 23:29:14.487633 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.487740 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":439,"threads_started":1}
I20260620 23:29:14.490625 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:14.489637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:50099)
0620 23:29:14.490223 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.490351 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":308,"threads_started":1}
I20260620 23:29:14.493541 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:13.488325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:51523)
0620 23:29:13.489058 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:14.493340 (+1004282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":451,"threads_started":1}
W20260620 23:29:14.494020 7552 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:45755 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:15.490633 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.489182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:38835)
0620 23:29:15.490324 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.490462 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":417,"threads_started":1}
I20260620 23:29:15.492825 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.491668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:33751)
0620 23:29:15.492434 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.492556 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":650,"threads_started":1}
I20260620 23:29:15.495543 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.494773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:58953)
0620 23:29:15.495236 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.495330 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260620 23:29:15.556753 7919 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:15.557083 7919 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:15.557791 7919 leader_election.cc:288] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755), 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252:34727)
I20260620 23:29:15.559248 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.558102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33626)
0620 23:29:15.558791 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.558982 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
I20260620 23:29:15.559278 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "643e504e8b2d4db28aeee4b200eb73eb" is_pre_election: true
I20260620 23:29:15.559710 7379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffd4a7787ced42269d1b4f3d43cd9772 in term 2.
I20260620 23:29:15.560432 7159 leader_election.cc:302] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643e504e8b2d4db28aeee4b200eb73eb, ffd4a7787ced42269d1b4f3d43cd9772; no voters:
I20260620 23:29:15.560796 7919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 23:29:15.560937 7919 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:15.561077 7919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:29:15.561540 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.561028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33640)
0620 23:29:15.561280 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.561387 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260620 23:29:15.561821 7159 leader_election.cc:334] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:15.563750 7919 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:15.564396 7919 leader_election.cc:288] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 3 election: Requested vote from peers 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755), 643e504e8b2d4db28aeee4b200eb73eb (127.0.96.252:34727)
I20260620 23:29:15.565282 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.564618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33650)
0620 23:29:15.564906 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.565068 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 23:29:15.565426 7379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "643e504e8b2d4db28aeee4b200eb73eb"
I20260620 23:29:15.565707 7379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:29:15.566452 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.566084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33654)
0620 23:29:15.566273 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.566343 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 23:29:15.566622 7159 leader_election.cc:334] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
I20260620 23:29:15.568094 7379 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffd4a7787ced42269d1b4f3d43cd9772 in term 3.
I20260620 23:29:15.568682 7159 leader_election.cc:302] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 643e504e8b2d4db28aeee4b200eb73eb, ffd4a7787ced42269d1b4f3d43cd9772; no voters: 36b55db1ddc34f209ac28fe8639a9a08
I20260620 23:29:15.569298 7919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:29:15.569885 7919 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [term 3 LEADER]: Becoming Leader. State: Replica: ffd4a7787ced42269d1b4f3d43cd9772, State: Running, Role: LEADER
I20260620 23:29:15.570389 7919 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } }
I20260620 23:29:15.572621 7922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd4a7787ced42269d1b4f3d43cd9772. Latest consensus state: current_term: 3 leader_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:15.573088 7922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:15.573949 7924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:15.578734 7924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:15.579773 7924 catalog_manager.cc:1259] Loaded cluster ID: 0403415ccf4e4c6fa5598b2f045fb462
I20260620 23:29:15.579864 7924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:15.581247 7924 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:15.582154 7924 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Loaded TSK: 0
I20260620 23:29:15.583273 7924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:29:15.623507 7173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:29:15.631753 7173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:15.637040 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.636560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33670)
0620 23:29:15.636741 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.636822 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:29:15.637328 7379 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [term 3 FOLLOWER]: Refusing update from remote peer ffd4a7787ced42269d1b4f3d43cd9772: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 23:29:15.638332 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.637870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33674)
0620 23:29:15.638121 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.638200 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:29:15.638429 7922 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 23:29:15.638921 7159 consensus_peers.cc:487] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 -> Peer 36b55db1ddc34f209ac28fe8639a9a08 (127.0.96.254:45755): Couldn't send request to peer 36b55db1ddc34f209ac28fe8639a9a08. Status: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 23:29:15.642163 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd4a7787ced42269d1b4f3d43cd9772. Latest consensus state: current_term: 3 leader_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:15.642514 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:15.644542 7919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:15.644930 7919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:15.646562 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "36b55db1ddc34f209ac28fe8639a9a08" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 45755 } } peers { permanent_uuid: "ffd4a7787ced42269d1b4f3d43cd9772" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 34031 } attrs { promote: false } } peers { permanent_uuid: "643e504e8b2d4db28aeee4b200eb73eb" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 34727 } attrs { promote: false } } }
I20260620 23:29:15.647171 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643e504e8b2d4db28aeee4b200eb73eb [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:29:15.664789 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.655054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:43425 (local address 127.0.0.1:40792)
0620 23:29:15.655268 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.655288 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:15.655417 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:15.656376 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:15.656383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:15.656429 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:15.656775 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.656791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.659068 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.659077 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.661660 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.661680 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.663771 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.663780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.664034 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.664091 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 23:29:15.664168 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 23:29:15.664358 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 23:29:15.666244 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.655267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.0.96.194:43425)
0620 23:29:15.656033 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:15.656041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:15.656063 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:29:15.656136 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:15.656140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:15.656213 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:15.656929 (+ 716us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:15.657173 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.658891 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.661927 (+ 3036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.663530 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.665426 (+ 1896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.665575 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:15.665635 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 23:29:15.665815 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":479,"threads_started":1}
I20260620 23:29:15.675352 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.655614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36653 (local address 127.0.0.1:60692)
0620 23:29:15.665085 (+ 9471us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.665109 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:15.665229 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:15.667446 (+ 2217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:15.667453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:15.667488 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:15.667714 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.667727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.669629 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.669637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.672090 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.672112 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.674291 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.674302 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.674625 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.674685 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 23:29:15.674770 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 23:29:15.674930 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9319,"threads_started":1}
I20260620 23:29:15.676302 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.656134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60692 (local address 127.0.96.193:36653)
0620 23:29:15.667014 (+ 10880us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:15.667023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:15.667049 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:29:15.667130 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:15.667139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:15.667254 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:15.667858 (+ 604us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:15.668068 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.669453 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.672363 (+ 2910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.674046 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.675623 (+ 1577us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.675736 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:15.675813 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:29:15.675947 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10698,"thread_start_us":10563,"threads_started":1}
I20260620 23:29:15.677853 7486 tablet_service.cc:1467] Processing CreateTablet for tablet 4466c92896ec4740bf4f71a16d83439b (DEFAULT_TABLE table=test-table [id=9503fc173eb2436da3641ebb285bd14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:15.677950 7626 tablet_service.cc:1467] Processing CreateTablet for tablet 4466c92896ec4740bf4f71a16d83439b (DEFAULT_TABLE table=test-table [id=9503fc173eb2436da3641ebb285bd14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:15.678349 7626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4466c92896ec4740bf4f71a16d83439b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:15.678349 7486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4466c92896ec4740bf4f71a16d83439b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:15.686134 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.655918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:34527 (local address 127.0.0.1:53312)
0620 23:29:15.675113 (+ 19195us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.675145 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:15.675445 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:15.676173 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:15.676178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:15.676209 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:15.676699 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.676713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.681944 (+ 5231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.681950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.683708 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.683725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.685115 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.685122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.685356 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.685403 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 23:29:15.685517 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 23:29:15.685685 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18994,"thread_start_us":18785,"threads_started":1}
I20260620 23:29:15.687116 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.656272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53312 (local address 127.0.96.195:34527)
0620 23:29:15.674256 (+ 17984us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:15.674268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:15.675330 (+ 1062us) server_negotiation.cc:407] Connection header received
0620 23:29:15.675854 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:15.675860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:15.675956 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:15.676075 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:15.676916 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.681725 (+ 4809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.683916 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.684831 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.685046 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.685202 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:15.686449 (+ 1247us) server_negotiation.cc:299] Negotiation successful
0620 23:29:15.686667 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17698,"thread_start_us":17450,"threads_started":1}
I20260620 23:29:15.687391 7938 tablet_bootstrap.cc:492] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Bootstrap starting.
I20260620 23:29:15.687865 7937 tablet_bootstrap.cc:492] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Bootstrap starting.
I20260620 23:29:15.688803 7767 tablet_service.cc:1467] Processing CreateTablet for tablet 4466c92896ec4740bf4f71a16d83439b (DEFAULT_TABLE table=test-table [id=9503fc173eb2436da3641ebb285bd14a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:15.689436 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4466c92896ec4740bf4f71a16d83439b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:15.689920 7938 tablet_bootstrap.cc:654] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:15.693264 7937 tablet_bootstrap.cc:654] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:15.695304 7938 tablet_bootstrap.cc:492] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: No bootstrap required, opened a new log
I20260620 23:29:15.695511 7938 ts_tablet_manager.cc:1397] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260620 23:29:15.696703 7937 tablet_bootstrap.cc:492] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: No bootstrap required, opened a new log
I20260620 23:29:15.696547 7938 raft_consensus.cc:348] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.696830 7938 raft_consensus.cc:374] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:15.696902 7937 ts_tablet_manager.cc:1397] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260620 23:29:15.696955 7938 raft_consensus.cc:729] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39b41dad4494e27a2076c4c322db8d5, State: Initialized, Role: FOLLOWER
I20260620 23:29:15.697412 7938 consensus_queue.cc:260] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.697741 7937 raft_consensus.cc:348] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.698005 7937 raft_consensus.cc:374] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:15.698123 7937 raft_consensus.cc:729] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9735cd69864afba13d4b00015f484c, State: Initialized, Role: FOLLOWER
I20260620 23:29:15.698365 7937 consensus_queue.cc:260] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.699452 7938 ts_tablet_manager.cc:1428] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 23:29:15.700934 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.700289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:49299)
0620 23:29:15.700543 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.700708 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:29:15.704084 7693 heartbeater.cc:503] Master 127.0.96.253:34031 was elected leader, sending a full tablet report...
I20260620 23:29:15.704385 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.702348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:44055)
0620 23:29:15.704010 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.704145 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1505}
I20260620 23:29:15.708185 7941 tablet_bootstrap.cc:492] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Bootstrap starting.
I20260620 23:29:15.708871 7937 ts_tablet_manager.cc:1428] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260620 23:29:15.710206 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.709615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:42887)
0620 23:29:15.709863 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.710009 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 23:29:15.711565 7941 tablet_bootstrap.cc:654] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:15.711755 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.711187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:46969)
0620 23:29:15.711388 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.711510 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 23:29:15.712519 7553 heartbeater.cc:503] Master 127.0.96.253:34031 was elected leader, sending a full tablet report...
I20260620 23:29:15.714272 7941 tablet_bootstrap.cc:492] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: No bootstrap required, opened a new log
I20260620 23:29:15.714478 7941 ts_tablet_manager.cc:1397] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20260620 23:29:15.715225 7941 raft_consensus.cc:348] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.715471 7941 raft_consensus.cc:374] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:15.715586 7941 raft_consensus.cc:729] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ddded06fcea475ea96243bb51197b85, State: Initialized, Role: FOLLOWER
I20260620 23:29:15.715910 7941 consensus_queue.cc:260] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.716959 7941 ts_tablet_manager.cc:1428] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 23:29:15.718175 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.717675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:52329)
0620 23:29:15.717848 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.717991 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:29:15.719069 7834 heartbeater.cc:503] Master 127.0.96.253:34031 was elected leader, sending a full tablet report...
I20260620 23:29:15.719636 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.719094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:40665)
0620 23:29:15.719289 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.719423 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:29:15.944546 7942 raft_consensus.cc:482] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:15.944772 7942 raft_consensus.cc:504] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.946637 7942 leader_election.cc:288] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653), 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527)
I20260620 23:29:15.955894 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.947289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:47037 (local address 127.0.96.193:36653)
0620 23:29:15.947499 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:15.947504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:15.947704 (+ 200us) server_negotiation.cc:407] Connection header received
0620 23:29:15.948047 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:15.948055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:15.948164 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:15.948296 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:15.948919 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.950134 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.952914 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.954323 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.954559 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.954655 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:15.955392 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 23:29:15.955563 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:29:15.955897 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.947183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:34047 (local address 127.0.96.195:34527)
0620 23:29:15.947396 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:15.947402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:15.948397 (+ 995us) server_negotiation.cc:407] Connection header received
0620 23:29:15.948612 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:15.948616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:15.948687 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:15.948773 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:15.949354 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.950381 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.952908 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.954323 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.954559 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.954655 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:15.955392 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 23:29:15.955564 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:29:15.956004 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.947091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36653 (local address 127.0.96.194:47037)
0620 23:29:15.947571 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.947593 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:15.947788 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:15.948346 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:15.948352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:15.948395 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:15.948697 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.948706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.950280 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.950284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.952651 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.952665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.954710 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.954715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.955028 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.955171 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 23:29:15.955299 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:29:15.955483 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":184,"spinlock_wait_cycles":12160}
I20260620 23:29:15.956035 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.947124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:34527 (local address 127.0.96.194:34047)
0620 23:29:15.948228 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.948243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:15.948392 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:15.948821 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:15.948826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:15.948852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:15.949198 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.949205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.950516 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.950521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.952655 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:15.952665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:15.954699 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:15.954707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:15.955031 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:15.955171 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 23:29:15.955332 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 23:29:15.955483 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":326,"spinlock_wait_cycles":768,"thread_start_us":512,"threads_started":1}
I20260620 23:29:15.957295 7506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4466c92896ec4740bf4f71a16d83439b" candidate_uuid: "d39b41dad4494e27a2076c4c322db8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9735cd69864afba13d4b00015f484c" is_pre_election: true
I20260620 23:29:15.957387 7787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4466c92896ec4740bf4f71a16d83439b" candidate_uuid: "d39b41dad4494e27a2076c4c322db8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ddded06fcea475ea96243bb51197b85" is_pre_election: true
I20260620 23:29:15.957835 7506 raft_consensus.cc:2454] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d39b41dad4494e27a2076c4c322db8d5 in term 0.
I20260620 23:29:15.958112 7787 raft_consensus.cc:2454] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d39b41dad4494e27a2076c4c322db8d5 in term 0.
I20260620 23:29:15.958704 7580 leader_election.cc:302] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9735cd69864afba13d4b00015f484c, d39b41dad4494e27a2076c4c322db8d5; no voters:
I20260620 23:29:15.959236 7942 raft_consensus.cc:2781] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:15.959386 7942 raft_consensus.cc:482] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:15.959498 7942 raft_consensus.cc:3037] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:15.961776 7942 raft_consensus.cc:504] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.962484 7942 leader_election.cc:288] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653), 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527)
I20260620 23:29:15.963060 7787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4466c92896ec4740bf4f71a16d83439b" candidate_uuid: "d39b41dad4494e27a2076c4c322db8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ddded06fcea475ea96243bb51197b85"
I20260620 23:29:15.963280 7787 raft_consensus.cc:3037] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:15.963164 7506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4466c92896ec4740bf4f71a16d83439b" candidate_uuid: "d39b41dad4494e27a2076c4c322db8d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9735cd69864afba13d4b00015f484c"
I20260620 23:29:15.963472 7506 raft_consensus.cc:3037] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:15.965617 7787 raft_consensus.cc:2454] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39b41dad4494e27a2076c4c322db8d5 in term 1.
I20260620 23:29:15.965962 7506 raft_consensus.cc:2454] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39b41dad4494e27a2076c4c322db8d5 in term 1.
I20260620 23:29:15.966163 7581 leader_election.cc:302] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ddded06fcea475ea96243bb51197b85, d39b41dad4494e27a2076c4c322db8d5; no voters:
I20260620 23:29:15.966573 7942 raft_consensus.cc:2781] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:15.967150 7942 raft_consensus.cc:686] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 1 LEADER]: Becoming Leader. State: Replica: d39b41dad4494e27a2076c4c322db8d5, State: Running, Role: LEADER
I20260620 23:29:15.967626 7942 consensus_queue.cc:237] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } }
I20260620 23:29:15.968519 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:15.967962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:44213)
0620 23:29:15.968151 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:15.968343 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 23:29:15.971990 7173 catalog_manager.cc:5495] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194). New cstate: current_term: 1 leader_uuid: "d39b41dad4494e27a2076c4c322db8d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39b41dad4494e27a2076c4c322db8d5" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 } health_report { overall_health: UNKNOWN } } }
I20260620 23:29:16.024590 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.023969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33684)
0620 23:29:16.024192 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.024288 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:29:16.026211 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.025596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:33694)
0620 23:29:16.025874 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.026001 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:29:16.059016 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.053907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.0.96.194:43425)
0620 23:29:16.054184 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:16.054189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:16.054204 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:29:16.054292 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:16.054296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:16.054371 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:16.054524 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:16.054880 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.055595 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.056887 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.057348 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.057485 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:16.057885 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:16.058437 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:16.058588 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:16.058665 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:29:16.058743 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:29:16.063951 7506 raft_consensus.cc:1261] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Refusing update from remote peer d39b41dad4494e27a2076c4c322db8d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:16.064275 7787 raft_consensus.cc:1261] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Refusing update from remote peer d39b41dad4494e27a2076c4c322db8d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:16.064954 7947 consensus_queue.cc:1035] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9735cd69864afba13d4b00015f484c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:16.065397 7942 consensus_queue.cc:1035] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ddded06fcea475ea96243bb51197b85" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 34527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:16.068697 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.068161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:37559)
0620 23:29:16.068424 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.068525 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 23:29:16.068807 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.068200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:36235)
0620 23:29:16.068466 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.068579 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 23:29:16.077106 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.076447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.195:55699)
0620 23:29:16.076712 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.076841 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 23:29:16.077765 7952 mvcc.cc:205] Tried to move back new op lower bound from 7299064447229042688 to 7299064446848479232. Current Snapshot: MvccSnapshot[applied={T|T < 7299064447229042688 or (T in {7299064447229042688})}]
I20260620 23:29:16.079876 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.077451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:54973)
0620 23:29:16.079599 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.079690 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2035}
I20260620 23:29:16.084631 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.084009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.194:40973)
0620 23:29:16.084266 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.084397 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 23:29:16.086108 7951 mvcc.cc:205] Tried to move back new op lower bound from 7299064447229042688 to 7299064446848479232. Current Snapshot: MvccSnapshot[applied={T|T < 7299064447229042688}]
I20260620 23:29:16.089242 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.088446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.96.193:34101)
0620 23:29:16.088843 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.088948 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260620 23:29:16.095496 7174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:16.095826 7174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:16.103266 7174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffd4a7787ced42269d1b4f3d43cd9772: Sending DeleteTablet for 3 replicas of tablet 4466c92896ec4740bf4f71a16d83439b
I20260620 23:29:16.104651 7626 tablet_service.cc:1514] Processing DeleteTablet for tablet 4466c92896ec4740bf4f71a16d83439b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:16 UTC) from {username='oryx', principal='oryx/127.0.96.253@KRBTEST.COM'} at 127.0.0.1:40792
I20260620 23:29:16.104809 7767 tablet_service.cc:1514] Processing DeleteTablet for tablet 4466c92896ec4740bf4f71a16d83439b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:16 UTC) from {username='oryx', principal='oryx/127.0.96.253@KRBTEST.COM'} at 127.0.0.1:53312
I20260620 23:29:16.104804 7486 tablet_service.cc:1514] Processing DeleteTablet for tablet 4466c92896ec4740bf4f71a16d83439b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:16 UTC) from {username='oryx', principal='oryx/127.0.96.253@KRBTEST.COM'} at 127.0.0.1:60692
I20260620 23:29:16.109072 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 7354
I20260620 23:29:16.117429 7959 tablet_replica.cc:330] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: stopping tablet replica
I20260620 23:29:16.117812 7959 raft_consensus.cc:2229] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:16.118192 7959 raft_consensus.cc:2258] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:16.120268 7959 ts_tablet_manager.cc:1905] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:16.124516 7961 tablet_replica.cc:330] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: stopping tablet replica
I20260620 23:29:16.124946 7961 raft_consensus.cc:2229] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:16.125814 7961 raft_consensus.cc:2258] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:16.126909 7960 tablet_replica.cc:330] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: stopping tablet replica
I20260620 23:29:16.127236 7960 raft_consensus.cc:2229] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:16.127617 7960 raft_consensus.cc:2258] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:16.127729 7961 ts_tablet_manager.cc:1905] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:16.129098 7960 ts_tablet_manager.cc:1905] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:16.131491 7959 ts_tablet_manager.cc:1918] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:16.131788 7959 log.cc:1196] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/4466c92896ec4740bf4f71a16d83439b
I20260620 23:29:16.132478 7959 ts_tablet_manager.cc:1939] T 4466c92896ec4740bf4f71a16d83439b P 1d9735cd69864afba13d4b00015f484c: Deleting consensus metadata
I20260620 23:29:16.134476 7159 catalog_manager.cc:4841] TS 1d9735cd69864afba13d4b00015f484c (127.0.96.193:36653): tablet 4466c92896ec4740bf4f71a16d83439b (table test-table [id=9503fc173eb2436da3641ebb285bd14a]) successfully deleted
I20260620 23:29:16.136323 7960 ts_tablet_manager.cc:1918] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:16.136521 7960 log.cc:1196] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/4466c92896ec4740bf4f71a16d83439b
I20260620 23:29:16.136992 7960 ts_tablet_manager.cc:1939] T 4466c92896ec4740bf4f71a16d83439b P 4ddded06fcea475ea96243bb51197b85: Deleting consensus metadata
I20260620 23:29:16.138990 7160 catalog_manager.cc:4841] TS 4ddded06fcea475ea96243bb51197b85 (127.0.96.195:34527): tablet 4466c92896ec4740bf4f71a16d83439b (table test-table [id=9503fc173eb2436da3641ebb285bd14a]) successfully deleted
I20260620 23:29:16.142657 7961 ts_tablet_manager.cc:1918] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:16.143119 7961 log.cc:1196] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/4466c92896ec4740bf4f71a16d83439b
I20260620 23:29:16.143906 7961 ts_tablet_manager.cc:1939] T 4466c92896ec4740bf4f71a16d83439b P d39b41dad4494e27a2076c4c322db8d5: Deleting consensus metadata
I20260620 23:29:16.145797 7159 catalog_manager.cc:4841] TS d39b41dad4494e27a2076c4c322db8d5 (127.0.96.194:43425): tablet 4466c92896ec4740bf4f71a16d83439b (table test-table [id=9503fc173eb2436da3641ebb285bd14a]) successfully deleted
I20260620 23:29:16.151808 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.111671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60700 (local address 127.0.96.193:36653)
0620 23:29:16.111975 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:16.111985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:16.119288 (+ 7303us) server_negotiation.cc:407] Connection header received
0620 23:29:16.119574 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:16.119579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:16.119673 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:16.119773 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:16.121883 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.123035 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.124804 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.125500 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.125621 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:16.143924 (+ 18303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:16.143947 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:16.143968 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:16.144020 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:16.148037 (+ 4017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:16.149315 (+ 1278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:16.149326 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:16.149330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:16.149499 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:16.150006 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:16.150010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:16.150014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:16.150471 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:16.150774 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:16.151272 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 23:29:16.151536 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:29:16.152668 7546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:16.250545 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 7354
I20260620 23:29:16.270198 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 7563
I20260620 23:29:16.285264 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.271213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.0.96.194:43425)
0620 23:29:16.271385 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:16.271390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:16.271571 (+ 181us) server_negotiation.cc:407] Connection header received
0620 23:29:16.271790 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:16.271796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:16.271882 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:16.272007 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:16.273334 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.274385 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.275831 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.276298 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.276460 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:16.279221 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:16.279243 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:16.279260 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:16.279298 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:16.282434 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:16.283119 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:16.283134 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:16.283136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:16.283286 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:16.283650 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:16.283655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:16.283658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:16.284024 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:16.284337 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:16.284658 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 23:29:16.284894 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 23:29:16.286478 7686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:16.420028 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 7563
I20260620 23:29:16.441860 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 7704
I20260620 23:29:16.462352 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.442731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.0.96.195:34527)
0620 23:29:16.442962 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:16.442970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:16.443062 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:29:16.443329 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:16.443333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:16.443406 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:16.443543 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:16.445085 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.445866 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.447323 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:16.447799 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:16.447960 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:16.455386 (+ 7426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:16.455417 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:16.455439 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:16.455494 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:16.458764 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:16.459704 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:16.459719 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:16.459722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:16.459875 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:16.460289 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:16.460297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:16.460301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:16.460784 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:16.461536 (+ 752us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:16.461690 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:29:16.461975 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:29:16.463199 7827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:16.533182 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.532200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:52338)
0620 23:29:16.532803 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.532905 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":343,"threads_started":1}
I20260620 23:29:16.534763 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:16.534389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:45755 (local address 127.0.0.1:52352)
0620 23:29:16.534548 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:16.534629 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 23:29:16.586046 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 7704
I20260620 23:29:16.612407 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 7028
I20260620 23:29:16.613487 7204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:16.735529 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 7028
W20260620 23:29:16.757515 7355 connection.cc:535] server connection from 127.0.0.1:55566 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:29:16.757826 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 7128
I20260620 23:29:16.759095 7399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:16.874899 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 7128
2026-06-20T23:29:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12481 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:16 dist-test-slave-v34r krb5kdc[7972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:16 dist-test-slave-v34r krb5kdc[7972](info): set up 2 sockets
Jun 20 23:29:16 dist-test-slave-v34r krb5kdc[7972](info): commencing operation
krb5kdc: starting...
W20260620 23:29:18.002086 387 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:18 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:18Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab.
Entry for principal oryx/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab.
I20260620 23:29:18.265179 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:39409
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab
--principal=oryx/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:39409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:18.669128 7985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:18.669600 7985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:18.669799 7985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:18.678122 7985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:18.678330 7985 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 23:29:18.678434 7985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:18.678557 7985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:18.678656 7985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:18.678752 7985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:18.690876 7985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:39409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.254.keytab
--principal=oryx/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:39409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:18.692402 7985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:18.693912 7985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:18.707940 7990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:18.707852 7991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:18.708379 7993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:18.708688 7985 server_base.cc:1034] running on GCE node
I20260620 23:29:18.709735 7985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:18.711229 7985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:18.712467 7985 hybrid_clock.cc:648] HybridClock initialized: now 1781998158712407 us; error 44 us; skew 500 ppm
I20260620 23:29:18.712850 7985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:18 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:18.721453 7985 init.cc:377] Logged in from keytab as oryx/127.0.96.254@KRBTEST.COM (short username oryx)
I20260620 23:29:18.723366 7985 webserver.cc:460] Webserver started at http://127.0.96.254:33163/ using document root <none> and password file <none>
I20260620 23:29:18.724150 7985 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:18.724273 7985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:18.724642 7985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:18.727773 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "37ca96b109df42008436fa0d06c69111"
format_stamp: "Formatted at 2026-06-20 23:29:18 on dist-test-slave-v34r"
server_key: "e676ad02017d9113802801f9f482fbbc"
server_key_iv: "ffac6ac578260fe5ff3372e6455b05eb"
server_key_version: "encryptionkey@0"
I20260620 23:29:18.728595 7985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "37ca96b109df42008436fa0d06c69111"
format_stamp: "Formatted at 2026-06-20 23:29:18 on dist-test-slave-v34r"
server_key: "e676ad02017d9113802801f9f482fbbc"
server_key_iv: "ffac6ac578260fe5ff3372e6455b05eb"
server_key_version: "encryptionkey@0"
I20260620 23:29:18.734484 7985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260620 23:29:18.738489 8000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:18.740325 7985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:29:18.740547 7985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "37ca96b109df42008436fa0d06c69111"
format_stamp: "Formatted at 2026-06-20 23:29:18 on dist-test-slave-v34r"
server_key: "e676ad02017d9113802801f9f482fbbc"
server_key_iv: "ffac6ac578260fe5ff3372e6455b05eb"
server_key_version: "encryptionkey@0"
I20260620 23:29:18.740780 7985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:18.762209 7985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:18.766531 7985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:18.766824 7985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:18.792270 7985 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:39409
I20260620 23:29:18.792435 8061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:39409 every 8 connection(s)
I20260620 23:29:18.793799 7985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:18.798694 8062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:18.803054 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 7985
I20260620 23:29:18.803283 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:18.803943 387 external_mini_cluster.cc:1428] Setting key cc5c87282b57bb39aa022bd3dea8d196
I20260620 23:29:18.815852 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Bootstrap starting.
I20260620 23:29:18.821322 8062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Neither blocks nor log segments found. Creating new log.
Jun 20 23:29:18 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:18.823179 8062 log.cc:824] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:18.827308 8062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: No bootstrap required, opened a new log
I20260620 23:29:18.834391 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:18.806542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50710 (local address 127.0.96.254:39409)
0620 23:29:18.807588 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:18.807608 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:18.807690 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:29:18.810200 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:18.810251 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:18.811022 (+ 771us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:18.811822 (+ 800us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:18.813260 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:18.815382 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:18.816834 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:18.818420 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:18.818767 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:18.823734 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:18.823746 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:18.823767 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:18.823828 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:18.828272 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:18.829473 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:18.829486 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:18.829496 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:18.829767 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:18.830270 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:18.830274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:18.830276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:18.831016 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:18.831315 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:18.831878 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 23:29:18.832651 (+ 773us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":410,"threads_started":1}
I20260620 23:29:18.838135 8062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ca96b109df42008436fa0d06c69111" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39409 } }
I20260620 23:29:18.838708 8062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:18.838862 8062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ca96b109df42008436fa0d06c69111, State: Initialized, Role: FOLLOWER
I20260620 23:29:18.839495 8062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ca96b109df42008436fa0d06c69111" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39409 } }
I20260620 23:29:18.839767 8062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:18.839929 8062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:18.840119 8062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:18.842661 8062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ca96b109df42008436fa0d06c69111" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39409 } }
I20260620 23:29:18.843273 8062 leader_election.cc:302] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37ca96b109df42008436fa0d06c69111; no voters:
I20260620 23:29:18.844017 8062 leader_election.cc:288] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:18.844226 8067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:18.845459 8067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [term 1 LEADER]: Becoming Leader. State: Replica: 37ca96b109df42008436fa0d06c69111, State: Running, Role: LEADER
I20260620 23:29:18.846151 8067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ca96b109df42008436fa0d06c69111" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39409 } }
I20260620 23:29:18.846544 8062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:18.850064 8069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37ca96b109df42008436fa0d06c69111. Latest consensus state: current_term: 1 leader_uuid: "37ca96b109df42008436fa0d06c69111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ca96b109df42008436fa0d06c69111" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39409 } } }
I20260620 23:29:18.850198 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37ca96b109df42008436fa0d06c69111" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ca96b109df42008436fa0d06c69111" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39409 } } }
I20260620 23:29:18.850677 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:18.850526 8069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:18.851861 8073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:18.859308 8073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:18.870829 8073 catalog_manager.cc:1347] Generated new cluster ID: 86dbf01fc9514c09b107d04151a4182f
I20260620 23:29:18.870990 8073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:18.908454 8073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:18.909981 8073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:18.917114 8073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Generated new TSK 0
I20260620 23:29:18.918046 8073 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab.
Entry for principal oryx/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab.
I20260620 23:29:18.970814 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39409
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:19.356065 8088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:19.356370 8088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:19.356528 8088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:19.364861 8088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:19.365037 8088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:19.365181 8088 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:29:19.365322 8088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:19.375645 8088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:39409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:19.377087 8088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:19.378427 8088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:19.393249 8094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:19.393262 8093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:19.394109 8096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:19.394657 8088 server_base.cc:1034] running on GCE node
I20260620 23:29:19.395578 8088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:19.396869 8088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:19.398123 8088 hybrid_clock.cc:648] HybridClock initialized: now 1781998159398025 us; error 68 us; skew 500 ppm
I20260620 23:29:19.398535 8088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:19 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:19.402554 8088 init.cc:377] Logged in from keytab as oryx/127.0.96.193@KRBTEST.COM (short username oryx)
I20260620 23:29:19.404506 8088 webserver.cc:460] Webserver started at http://127.0.96.193:46283/ using document root <none> and password file <none>
I20260620 23:29:19.405509 8088 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:19.405618 8088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:19.405961 8088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:19.409682 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "bc295c278d844c558890837e59b3cf2d"
format_stamp: "Formatted at 2026-06-20 23:29:19 on dist-test-slave-v34r"
server_key: "3b5361484674150135f47777e49fe54a"
server_key_iv: "374a4b2a427b25b89b406e12be62d47b"
server_key_version: "encryptionkey@0"
I20260620 23:29:19.410661 8088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "bc295c278d844c558890837e59b3cf2d"
format_stamp: "Formatted at 2026-06-20 23:29:19 on dist-test-slave-v34r"
server_key: "3b5361484674150135f47777e49fe54a"
server_key_iv: "374a4b2a427b25b89b406e12be62d47b"
server_key_version: "encryptionkey@0"
I20260620 23:29:19.417390 8088 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 23:29:19.421954 8103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:19.423784 8088 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:29:19.424119 8088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "bc295c278d844c558890837e59b3cf2d"
format_stamp: "Formatted at 2026-06-20 23:29:19 on dist-test-slave-v34r"
server_key: "3b5361484674150135f47777e49fe54a"
server_key_iv: "374a4b2a427b25b89b406e12be62d47b"
server_key_version: "encryptionkey@0"
I20260620 23:29:19.424430 8088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:19.454355 8088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:19.459842 8088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:19.460193 8088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:19.463626 8088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:19.463762 8088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:19.463959 8088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:19.464058 8088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:29:19 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:19.492286 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:19.464250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:45981 (local address 127.0.96.254:39409)
0620 23:29:19.464840 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:19.464848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:19.466502 (+ 1654us) server_negotiation.cc:407] Connection header received
0620 23:29:19.468954 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:19.468963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:19.469131 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:19.469318 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:19.472090 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.473359 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.476385 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.476806 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.476951 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:19.482406 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:19.482419 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:19.482428 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:19.482505 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:19.486826 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:19.488424 (+ 1598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:19.488435 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:19.488441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:19.488653 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:19.489528 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:19.489536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:19.489542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:19.490050 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:19.490221 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:19.491817 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0620 23:29:19.492021 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":364,"threads_started":1}
I20260620 23:29:19.495252 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:19.465332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.96.193:45981)
0620 23:29:19.466296 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:19.466352 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:19.468467 (+ 2115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:19.469645 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:19.469657 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:19.470576 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:19.471780 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:19.471812 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.473543 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.473547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:19.476229 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:19.476240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.477110 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.477117 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:19.477403 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:19.478336 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:19.478377 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:19.481984 (+ 3607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:19.487069 (+ 5085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:19.487082 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:19.487111 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:19.488140 (+ 1029us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:19.488854 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:19.488863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:19.488870 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:19.489335 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:19.490340 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:19.490351 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:19.491135 (+ 784us) client_negotiation.cc:769] Sending connection context
0620 23:29:19.491610 (+ 475us) client_negotiation.cc:240] Negotiation successful
0620 23:29:19.492555 (+ 945us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":398,"threads_started":1}
I20260620 23:29:19.519155 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:19.502683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.96.193:33097)
0620 23:29:19.505667 (+ 2984us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:19.505693 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:19.505880 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:19.506235 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:19.506239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:19.506735 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:19.507034 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:19.507043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.508021 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.508026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:19.509132 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:19.509143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.509893 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.509900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:19.510079 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:19.511094 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:19.511110 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:19.512371 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:19.516226 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:19.516232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:19.516236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:19.516680 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:19.517146 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:19.517152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:19.517156 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:19.517291 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:19.517978 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:19.517983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:19.518262 (+ 279us) client_negotiation.cc:769] Sending connection context
0620 23:29:19.518477 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 23:29:19.518817 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2814,"thread_start_us":2570,"threads_started":1}
I20260620 23:29:19.519744 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:19.502782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:33097 (local address 127.0.96.254:39409)
0620 23:29:19.503068 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:19.503080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:19.505797 (+ 2717us) server_negotiation.cc:407] Connection header received
0620 23:29:19.506031 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:19.506035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:19.506100 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:19.506210 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:19.507176 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.507866 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.509265 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:19.509682 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:19.509808 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:19.512645 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:19.512657 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:19.512660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:19.512689 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:19.516076 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:19.516830 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:19.516836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:19.516840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:19.517027 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:19.517411 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:19.517418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:19.517424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:19.517862 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:19.518029 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:19.518716 (+ 687us) server_negotiation.cc:299] Negotiation successful
0620 23:29:19.519163 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:29:19.531590 8088 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:38855
I20260620 23:29:19.531626 8223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:38855 every 8 connection(s)
I20260620 23:29:19.533267 8088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:19.535856 8224 heartbeater.cc:347] Connected to a master server at 127.0.96.254:39409
I20260620 23:29:19.536194 8224 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:19.536417 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 8088
I20260620 23:29:19.536650 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:19.537356 8224 heartbeater.cc:511] Master 127.0.96.254:39409 requested a full tablet report, sending...
I20260620 23:29:19.537603 387 external_mini_cluster.cc:1428] Setting key 11794b626c5e3f2b1fde5d5dceb5cf60
I20260620 23:29:19.540601 8016 ts_manager.cc:194] Registered new tserver with Master: bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855)
I20260620 23:29:19.543190 8016 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:45981
WARNING: no policy specified for oryx/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab.
Entry for principal oryx/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab.
I20260620 23:29:19.594650 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39409
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:19.935333 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:19.935631 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:19.935761 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:19.943397 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:19.943521 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:19.943637 8228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:29:19.943799 8228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:19.954375 8228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:39409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:19.955741 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:19.957134 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:19.972215 8234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:19.972438 8236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:19.972261 8233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:19.973222 8228 server_base.cc:1034] running on GCE node
I20260620 23:29:19.974018 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:19.975061 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:19.976359 8228 hybrid_clock.cc:648] HybridClock initialized: now 1781998159976257 us; error 79 us; skew 500 ppm
I20260620 23:29:19.976715 8228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:19 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:19.980414 8228 init.cc:377] Logged in from keytab as oryx/127.0.96.194@KRBTEST.COM (short username oryx)
I20260620 23:29:19.982331 8228 webserver.cc:460] Webserver started at http://127.0.96.194:43009/ using document root <none> and password file <none>
I20260620 23:29:19.983287 8228 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:19.983392 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:19.983825 8228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:19.987568 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "b6275037127246ab967a45bb3c0bad45"
format_stamp: "Formatted at 2026-06-20 23:29:19 on dist-test-slave-v34r"
server_key: "eb17da1f069032ffa01df8dddb334a73"
server_key_iv: "0e7f32b12b1b56238bb8a979bad3894b"
server_key_version: "encryptionkey@0"
I20260620 23:29:19.988571 8228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "b6275037127246ab967a45bb3c0bad45"
format_stamp: "Formatted at 2026-06-20 23:29:19 on dist-test-slave-v34r"
server_key: "eb17da1f069032ffa01df8dddb334a73"
server_key_iv: "0e7f32b12b1b56238bb8a979bad3894b"
server_key_version: "encryptionkey@0"
I20260620 23:29:19.995182 8228 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260620 23:29:19.999811 8243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:20.001859 8228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:20.002123 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "b6275037127246ab967a45bb3c0bad45"
format_stamp: "Formatted at 2026-06-20 23:29:19 on dist-test-slave-v34r"
server_key: "eb17da1f069032ffa01df8dddb334a73"
server_key_iv: "0e7f32b12b1b56238bb8a979bad3894b"
server_key_version: "encryptionkey@0"
I20260620 23:29:20.002453 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:20.053656 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:20.059298 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:20.059684 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:20.063236 8228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:20.063385 8228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:20.063549 8228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:20.063623 8228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998159, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:20.096503 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.064372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:50125 (local address 127.0.96.254:39409)
0620 23:29:20.064812 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.064816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.066595 (+ 1779us) server_negotiation.cc:407] Connection header received
0620 23:29:20.069559 (+ 2964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.069565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.069658 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.069802 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:20.074163 (+ 4361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.075577 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.080006 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.080540 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.080737 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.088043 (+ 7306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:20.088049 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:20.088053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:20.088093 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:20.091432 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.092705 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.092711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.092715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.092845 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.093618 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.093628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.093631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.094228 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:20.094476 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.095802 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.096093 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":243,"threads_started":1}
I20260620 23:29:20.097878 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.065389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.96.194:50125)
0620 23:29:20.066400 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.066459 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.069198 (+ 2739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.070412 (+ 1214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.070432 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.071739 (+ 1307us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:20.073743 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.073797 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.075938 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.075943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.079621 (+ 3678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.079661 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.080944 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.080957 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.081498 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.082723 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:20.082758 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:20.087575 (+ 4817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:20.091689 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.091701 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.091724 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.092520 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.093021 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.093027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.093031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.093374 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.094571 (+ 1197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:20.094601 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:20.095275 (+ 674us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.095674 (+ 399us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.096293 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":440,"threads_started":1}
I20260620 23:29:20.117467 8228 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:40535
I20260620 23:29:20.117537 8363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:40535 every 8 connection(s)
I20260620 23:29:20.118886 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:20.121908 8364 heartbeater.cc:347] Connected to a master server at 127.0.96.254:39409
I20260620 23:29:20.122232 8364 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:20.122886 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 8228
I20260620 23:29:20.123066 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:20.123150 8364 heartbeater.cc:511] Master 127.0.96.254:39409 requested a full tablet report, sending...
I20260620 23:29:20.123570 387 external_mini_cluster.cc:1428] Setting key c13df0352cba18d58a37d2f7f1196059
I20260620 23:29:20.125257 8017 ts_manager.cc:194] Registered new tserver with Master: b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:20.126636 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.109034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:38587 (local address 127.0.96.254:39409)
0620 23:29:20.109296 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.109306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.110024 (+ 718us) server_negotiation.cc:407] Connection header received
0620 23:29:20.110316 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.110324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.110447 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.110613 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:20.111805 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.113089 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.115639 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.116418 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.116770 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.119511 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:20.119520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:20.119529 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:20.119585 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:20.123109 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.124283 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.124287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.124290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.124418 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.124876 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.124881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.124883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.125379 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:20.125601 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.126152 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.126342 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:29:20.126825 8017 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:50125
I20260620 23:29:20.126948 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.109031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.96.194:38587)
0620 23:29:20.109857 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.109882 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.110118 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.110737 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.110744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.111272 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:20.111622 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.111640 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.113296 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.113305 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.115442 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.115461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.116840 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.116847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.117145 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.118178 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:20.118200 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:20.119237 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:20.123262 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.123271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.123279 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.123905 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.124547 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.124556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.124562 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.124742 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.125580 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:20.125587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:20.125830 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.126012 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.126450 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":466,"threads_started":1}
WARNING: no policy specified for oryx/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab.
Entry for principal oryx/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab.
I20260620 23:29:20.181181 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39409
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:20.548278 8224 heartbeater.cc:503] Master 127.0.96.254:39409 was elected leader, sending a full tablet report...
W20260620 23:29:20.593711 8368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:20.594115 8368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:20.594228 8368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:20.603299 8368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:20.603473 8368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:20.603570 8368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 23:29:20.603664 8368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:20.615445 8368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.96.254:39409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:20.617236 8368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:20.618920 8368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:20.635996 8373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:20.635825 8376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:20.635825 8374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:20.636952 8368 server_base.cc:1034] running on GCE node
I20260620 23:29:20.637889 8368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:20.639205 8368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:20.640479 8368 hybrid_clock.cc:648] HybridClock initialized: now 1781998160640397 us; error 53 us; skew 500 ppm
I20260620 23:29:20.640911 8368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:20.645226 8368 init.cc:377] Logged in from keytab as oryx/127.0.96.195@KRBTEST.COM (short username oryx)
I20260620 23:29:20.647380 8368 webserver.cc:460] Webserver started at http://127.0.96.195:33931/ using document root <none> and password file <none>
I20260620 23:29:20.648353 8368 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:20.648473 8368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:20.648804 8368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:20.652484 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "674db3bc67e54fa7ad72b667bbce7da3"
format_stamp: "Formatted at 2026-06-20 23:29:20 on dist-test-slave-v34r"
server_key: "89334139d642d2158e085dce2c130e8c"
server_key_iv: "a9b97e08ddaaa83083e693ef1977c585"
server_key_version: "encryptionkey@0"
I20260620 23:29:20.653594 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "674db3bc67e54fa7ad72b667bbce7da3"
format_stamp: "Formatted at 2026-06-20 23:29:20 on dist-test-slave-v34r"
server_key: "89334139d642d2158e085dce2c130e8c"
server_key_iv: "a9b97e08ddaaa83083e693ef1977c585"
server_key_version: "encryptionkey@0"
I20260620 23:29:20.660475 8368 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:29:20.665053 8383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:20.666970 8368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:20.667227 8368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "674db3bc67e54fa7ad72b667bbce7da3"
format_stamp: "Formatted at 2026-06-20 23:29:20 on dist-test-slave-v34r"
server_key: "89334139d642d2158e085dce2c130e8c"
server_key_iv: "a9b97e08ddaaa83083e693ef1977c585"
server_key_version: "encryptionkey@0"
I20260620 23:29:20.667531 8368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:20.688423 8368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:20.693724 8368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:20.694080 8368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:20.697679 8368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:20.697806 8368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:20.697997 8368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:20.698089 8368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998160, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:20.728331 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.699336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.96.195:47439)
0620 23:29:20.700393 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.700452 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.703067 (+ 2615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.704316 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.704330 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.705344 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:20.706320 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.706351 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.708517 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.708536 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.711178 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.711190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.712184 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.712193 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.712615 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.713593 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:20.713616 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:20.718100 (+ 4484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:20.721690 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.721706 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.721747 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.722829 (+ 1082us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.723545 (+ 716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.723551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.723559 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.724022 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.724797 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:20.724810 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:20.725390 (+ 580us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.725813 (+ 423us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.726466 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":429,"threads_started":1}
I20260620 23:29:20.729481 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.698450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:47439 (local address 127.0.96.254:39409)
0620 23:29:20.699241 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.699245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.700607 (+ 1362us) server_negotiation.cc:407] Connection header received
0620 23:29:20.703395 (+ 2788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.703401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.703489 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.703591 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:20.706618 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.708248 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.711395 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.711842 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.711989 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.718547 (+ 6558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:20.718555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:20.718561 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:20.718611 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:20.721460 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.723098 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.723107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.723115 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.723338 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.724219 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.724227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.724234 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.724614 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:20.724783 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.728865 (+ 4082us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.729149 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":563,"threads_started":1}
I20260620 23:29:20.760623 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.737014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:45705 (local address 127.0.96.254:39409)
0620 23:29:20.737290 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.737299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.742182 (+ 4883us) server_negotiation.cc:407] Connection header received
0620 23:29:20.742645 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.742655 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.742791 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.742977 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:20.745157 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.746233 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.748915 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.749499 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.749666 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.751877 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:20.751885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:20.751895 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:20.751929 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:20.756455 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.757694 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.757701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.757706 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.757903 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.758520 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.758529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.758536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.759064 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:20.759307 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.759924 (+ 617us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.760215 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:29:20.760746 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.736314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.96.195:45705)
0620 23:29:20.742003 (+ 5689us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.742039 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.742371 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.743044 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.743052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.744269 (+ 1217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:20.744912 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.744927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.746485 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.746493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.748692 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.748706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.749781 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.749789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.749978 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.750793 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:20.750814 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:20.751636 (+ 822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:20.756677 (+ 5041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.756691 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.756694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.757435 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.758029 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:20.758035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:20.758043 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:20.758256 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:20.759246 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:20.759252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:20.759604 (+ 352us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.759791 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.760246 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5441,"thread_start_us":5173,"threads_started":1}
I20260620 23:29:20.767307 8368 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:36261
I20260620 23:29:20.767405 8503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:36261 every 8 connection(s)
I20260620 23:29:20.768743 8368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:20.771286 8504 heartbeater.cc:347] Connected to a master server at 127.0.96.254:39409
I20260620 23:29:20.771601 8504 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:20.772552 8504 heartbeater.cc:511] Master 127.0.96.254:39409 requested a full tablet report, sending...
I20260620 23:29:20.774310 8017 ts_manager.cc:194] Registered new tserver with Master: 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261)
I20260620 23:29:20.775326 8017 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:47439
I20260620 23:29:20.778571 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 8368
I20260620 23:29:20.778728 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:20.779196 387 external_mini_cluster.cc:1428] Setting key a3196b13fc68f83fa42277e4063924a6
I20260620 23:29:20.783109 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:29:20.843047 8392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:20.831256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.0.96.254:39409)
0620 23:29:20.831462 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.831467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.831604 (+ 137us) server_negotiation.cc:407] Connection header received
0620 23:29:20.831792 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.831797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.831905 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.832033 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:20.833512 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.834324 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.835715 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.836232 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.836496 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.842663 (+ 6167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50720: BlockingRecv error: recv got EOF from 127.0.0.1:50720 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:29:20 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:20.863366 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.847239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.96.254:39409)
0620 23:29:20.847537 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.847544 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.848241 (+ 697us) server_negotiation.cc:407] Connection header received
0620 23:29:20.848418 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.848423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.848507 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.848596 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:20.850019 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.850753 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.852194 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.852642 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.852756 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.857725 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:20.857731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:20.857735 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:20.857786 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:20.860471 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.861423 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.861427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.861430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.861562 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:20.861939 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:20.861942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:20.861944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:20.862194 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:20.862372 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.862863 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.863092 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 23:29:20.868492 8017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:29:20.876907 8017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:20.906270 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.893906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:40535 (local address 127.0.0.1:42236)
0620 23:29:20.895286 (+ 1380us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.895307 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.895476 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.896937 (+ 1461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.896943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.897002 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:20.897568 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.897580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.900008 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.900017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.903026 (+ 3009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.903045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.905169 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.905176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.905370 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.905423 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.905579 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.905771 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"mutex_wait_us":4,"thread_start_us":1254,"threads_started":1}
I20260620 23:29:20.907325 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.895011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.0.96.194:40535)
0620 23:29:20.896096 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.896108 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.896146 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:29:20.896278 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.896291 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.896655 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.897766 (+ 1111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:20.897959 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.899808 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.903243 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.904897 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.905123 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.905385 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.906776 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.907004 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":771,"threads_started":1}
I20260620 23:29:20.907680 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.892903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.0.96.195:36261)
0620 23:29:20.895132 (+ 2229us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.895147 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.895206 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:29:20.895354 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.895368 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.895742 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.897458 (+ 1716us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:20.898040 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.900582 (+ 2542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.903943 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.905541 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.905675 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.905911 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.907082 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.907321 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1972,"thread_start_us":1701,"threads_started":1}
I20260620 23:29:20.907972 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.892690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:36261 (local address 127.0.0.1:41738)
0620 23:29:20.893591 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.893635 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.893844 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.896092 (+ 2248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.896105 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.896151 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:20.896833 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.896851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.900838 (+ 3987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.900865 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.903710 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.903727 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.906510 (+ 2783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.906518 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.906721 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.906870 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.906996 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.907205 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":96,"thread_start_us":415,"threads_started":1}
I20260620 23:29:20.911903 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.894083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38855 (local address 127.0.0.1:33886)
0620 23:29:20.903046 (+ 8963us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.903066 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.903193 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.903784 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.903788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.903813 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:20.904212 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.904227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.906128 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.906133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.909041 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.909059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.910924 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.910931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.911129 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.911180 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.911291 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.911462 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8795,"mutex_wait_us":325,"thread_start_us":8649,"threads_started":1}
I20260620 23:29:20.912482 8434 tablet_service.cc:1467] Processing CreateTablet for tablet ec8c17be53694446ba1a246d5dc060d8 (DEFAULT_TABLE table=test-table [id=1714876d194b469f98d2777cac05b8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:20.913429 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.894609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.0.96.193:38855)
0620 23:29:20.902526 (+ 7917us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.902536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.903241 (+ 705us) server_negotiation.cc:407] Connection header received
0620 23:29:20.903359 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.903366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.903592 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.903803 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:20.904415 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.905910 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.909233 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.910653 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.910795 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.912778 (+ 1983us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.912855 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.913065 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7679,"thread_start_us":7592,"threads_started":1}
I20260620 23:29:20.912129 8294 tablet_service.cc:1467] Processing CreateTablet for tablet ec8c17be53694446ba1a246d5dc060d8 (DEFAULT_TABLE table=test-table [id=1714876d194b469f98d2777cac05b8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:20.914201 8434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8c17be53694446ba1a246d5dc060d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:20.914525 8294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8c17be53694446ba1a246d5dc060d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:20.917840 8154 tablet_service.cc:1467] Processing CreateTablet for tablet ec8c17be53694446ba1a246d5dc060d8 (DEFAULT_TABLE table=test-table [id=1714876d194b469f98d2777cac05b8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:20.919611 8154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec8c17be53694446ba1a246d5dc060d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:20.926131 8528 tablet_bootstrap.cc:492] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Bootstrap starting.
I20260620 23:29:20.930558 8528 tablet_bootstrap.cc:654] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:20.931330 8529 tablet_bootstrap.cc:492] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Bootstrap starting.
I20260620 23:29:20.932003 8528 log.cc:824] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:20.934993 8529 tablet_bootstrap.cc:654] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:20.935842 8528 tablet_bootstrap.cc:492] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: No bootstrap required, opened a new log
I20260620 23:29:20.936167 8528 ts_tablet_manager.cc:1397] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260620 23:29:20.936489 8531 tablet_bootstrap.cc:492] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Bootstrap starting.
I20260620 23:29:20.938292 8529 log.cc:824] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:20.940534 8531 tablet_bootstrap.cc:654] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:20.941846 8531 log.cc:824] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:20.941910 8529 tablet_bootstrap.cc:492] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: No bootstrap required, opened a new log
I20260620 23:29:20.942193 8529 ts_tablet_manager.cc:1397] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:29:20.945637 8531 tablet_bootstrap.cc:492] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: No bootstrap required, opened a new log
I20260620 23:29:20.946007 8531 ts_tablet_manager.cc:1397] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 23:29:20.946513 8528 raft_consensus.cc:348] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.946995 8528 raft_consensus.cc:374] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:20.947084 8528 raft_consensus.cc:729] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Initialized, Role: FOLLOWER
I20260620 23:29:20.947744 8528 consensus_queue.cc:260] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.949194 8504 heartbeater.cc:503] Master 127.0.96.254:39409 was elected leader, sending a full tablet report...
I20260620 23:29:20.949859 8528 ts_tablet_manager.cc:1428] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 23:29:20.953763 8529 raft_consensus.cc:348] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.954283 8529 raft_consensus.cc:374] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:20.954380 8529 raft_consensus.cc:729] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6275037127246ab967a45bb3c0bad45, State: Initialized, Role: FOLLOWER
I20260620 23:29:20.955044 8529 consensus_queue.cc:260] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.956197 8364 heartbeater.cc:503] Master 127.0.96.254:39409 was elected leader, sending a full tablet report...
I20260620 23:29:20.956351 8531 raft_consensus.cc:348] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.956954 8531 raft_consensus.cc:374] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:20.957135 8531 raft_consensus.cc:729] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc295c278d844c558890837e59b3cf2d, State: Initialized, Role: FOLLOWER
I20260620 23:29:20.958063 8531 consensus_queue.cc:260] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.958380 8529 ts_tablet_manager.cc:1428] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260620 23:29:20.960147 8531 ts_tablet_manager.cc:1428] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260620 23:29:20.961287 8535 raft_consensus.cc:482] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:20.961585 8535 raft_consensus.cc:504] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.963431 8535 leader_election.cc:288] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261), bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855)
I20260620 23:29:20.971210 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.963856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:41053 (local address 127.0.96.195:36261)
0620 23:29:20.964048 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.964052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.964731 (+ 679us) server_negotiation.cc:407] Connection header received
0620 23:29:20.965076 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.965083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.965161 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.965229 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:20.966040 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.967166 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.969244 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.970085 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.970323 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.970416 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.970744 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.970889 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 23:29:20.971247 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.965150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:44851 (local address 127.0.96.193:38855)
0620 23:29:20.965337 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:20.965343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:20.965750 (+ 407us) server_negotiation.cc:407] Connection header received
0620 23:29:20.965909 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:20.965917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:20.966008 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:20.966099 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:20.966760 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.967649 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.969547 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.970303 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.970490 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.970570 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:20.970853 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 23:29:20.970989 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:29:20.971424 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.963864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:36261 (local address 127.0.96.194:41053)
0620 23:29:20.964634 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.964656 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.964845 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.965457 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.965461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.965481 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:20.965860 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.965874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.967310 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.967317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.968969 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.969061 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.970346 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.970352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.970556 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.970586 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.970714 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.970881 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":1,"thread_start_us":423,"threads_started":1}
I20260620 23:29:20.971470 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:20.964887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38855 (local address 127.0.96.194:44851)
0620 23:29:20.965617 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:20.965638 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:20.965783 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:20.966141 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:20.966149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:20.966176 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:20.966593 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.966605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.967803 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.967809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.969403 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:20.969416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:20.970547 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:20.970553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:20.970698 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:20.970724 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:29:20.970833 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:29:20.971015 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":365,"threads_started":1}
I20260620 23:29:20.972970 8458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c17be53694446ba1a246d5dc060d8" candidate_uuid: "b6275037127246ab967a45bb3c0bad45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "674db3bc67e54fa7ad72b667bbce7da3" is_pre_election: true
I20260620 23:29:20.973039 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c17be53694446ba1a246d5dc060d8" candidate_uuid: "b6275037127246ab967a45bb3c0bad45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d" is_pre_election: true
I20260620 23:29:20.973591 8458 raft_consensus.cc:2454] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6275037127246ab967a45bb3c0bad45 in term 0.
I20260620 23:29:20.973712 8174 raft_consensus.cc:2454] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6275037127246ab967a45bb3c0bad45 in term 0.
I20260620 23:29:20.974385 8244 leader_election.cc:302] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, b6275037127246ab967a45bb3c0bad45; no voters:
I20260620 23:29:20.974823 8535 raft_consensus.cc:2781] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:20.975013 8535 raft_consensus.cc:482] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:20.975184 8535 raft_consensus.cc:3037] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:20.977774 8535 raft_consensus.cc:504] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.978488 8535 leader_election.cc:288] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [CANDIDATE]: Term 1 election: Requested vote from peers 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261), bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855)
I20260620 23:29:20.979151 8458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c17be53694446ba1a246d5dc060d8" candidate_uuid: "b6275037127246ab967a45bb3c0bad45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "674db3bc67e54fa7ad72b667bbce7da3"
I20260620 23:29:20.979212 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec8c17be53694446ba1a246d5dc060d8" candidate_uuid: "b6275037127246ab967a45bb3c0bad45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d"
I20260620 23:29:20.979385 8458 raft_consensus.cc:3037] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:20.979445 8174 raft_consensus.cc:3037] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:20.982745 8174 raft_consensus.cc:2454] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6275037127246ab967a45bb3c0bad45 in term 1.
I20260620 23:29:20.982745 8458 raft_consensus.cc:2454] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6275037127246ab967a45bb3c0bad45 in term 1.
I20260620 23:29:20.983319 8244 leader_election.cc:302] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b6275037127246ab967a45bb3c0bad45, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:20.983768 8535 raft_consensus.cc:2781] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:20.984328 8535 raft_consensus.cc:686] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 1 LEADER]: Becoming Leader. State: Replica: b6275037127246ab967a45bb3c0bad45, State: Running, Role: LEADER
I20260620 23:29:20.984851 8535 consensus_queue.cc:237] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } }
I20260620 23:29:20.989981 8016 catalog_manager.cc:5495] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6275037127246ab967a45bb3c0bad45 (127.0.96.194). New cstate: current_term: 1 leader_uuid: "b6275037127246ab967a45bb3c0bad45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } health_report { overall_health: HEALTHY } } }
I20260620 23:29:21.014019 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.007785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.0.96.194:40535)
0620 23:29:21.008097 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.008101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.008245 (+ 144us) server_negotiation.cc:407] Connection header received
0620 23:29:21.008509 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.008514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.008597 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.008780 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:21.009298 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.010061 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.011562 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.012000 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.012121 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.012679 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:21.013371 (+ 692us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:21.013489 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.013690 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.013760 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
W20260620 23:29:21.020623 8505 tablet.cc:2369] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:21.024408 8458 raft_consensus.cc:1261] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Refusing update from remote peer b6275037127246ab967a45bb3c0bad45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:21.024425 8174 raft_consensus.cc:1261] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Refusing update from remote peer b6275037127246ab967a45bb3c0bad45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:21.025736 8539 consensus_queue.cc:1035] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:21.026223 8535 consensus_queue.cc:1035] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:29:21.036412 8225 tablet.cc:2369] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:21.045048 8544 mvcc.cc:205] Tried to move back new op lower bound from 7299064467545153536 to 7299064467397914624. Current Snapshot: MvccSnapshot[applied={T|T < 7299064467545153536}]
I20260620 23:29:21.059767 8016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:21.060160 8016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:21.064541 8016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Sending DeleteTablet for 3 replicas of tablet ec8c17be53694446ba1a246d5dc060d8
I20260620 23:29:21.065944 8434 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8c17be53694446ba1a246d5dc060d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:21 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:41738
I20260620 23:29:21.066073 8294 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8c17be53694446ba1a246d5dc060d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:21 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42236
I20260620 23:29:21.066416 8154 tablet_service.cc:1514] Processing DeleteTablet for tablet ec8c17be53694446ba1a246d5dc060d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:21 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:33886
I20260620 23:29:21.069602 8016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:29:21.071540 8016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:21.084373 8551 tablet_replica.cc:330] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: stopping tablet replica
I20260620 23:29:21.084832 8293 tablet_service.cc:1467] Processing CreateTablet for tablet ed1bea07282f4e4bbd1b6538a1abc3b4 (DEFAULT_TABLE table=test-table [id=2507b81564094564adde650d2e0b3420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:21.085095 8551 raft_consensus.cc:2229] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:21.085510 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1bea07282f4e4bbd1b6538a1abc3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.085743 8551 raft_consensus.cc:2258] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:21.087205 8552 tablet_replica.cc:330] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: stopping tablet replica
I20260620 23:29:21.087853 8552 raft_consensus.cc:2229] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:21.088404 8551 ts_tablet_manager.cc:1905] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:21.090471 8153 tablet_service.cc:1467] Processing CreateTablet for tablet ed1bea07282f4e4bbd1b6538a1abc3b4 (DEFAULT_TABLE table=test-table [id=2507b81564094564adde650d2e0b3420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:21.091042 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1bea07282f4e4bbd1b6538a1abc3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.092139 8553 tablet_replica.cc:330] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: stopping tablet replica
I20260620 23:29:21.092347 8434 tablet_service.cc:1467] Processing CreateTablet for tablet ed1bea07282f4e4bbd1b6538a1abc3b4 (DEFAULT_TABLE table=test-table [id=2507b81564094564adde650d2e0b3420]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:21.092839 8434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed1bea07282f4e4bbd1b6538a1abc3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.092775 8553 raft_consensus.cc:2229] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:21.093505 8553 raft_consensus.cc:2258] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:21.094236 8552 raft_consensus.cc:2258] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:21.097671 8529 tablet_bootstrap.cc:492] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Bootstrap starting.
I20260620 23:29:21.098245 8551 ts_tablet_manager.cc:1918] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:21.098506 8551 log.cc:1196] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/ec8c17be53694446ba1a246d5dc060d8
I20260620 23:29:21.099210 8551 ts_tablet_manager.cc:1939] T ec8c17be53694446ba1a246d5dc060d8 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting consensus metadata
I20260620 23:29:21.099606 8529 tablet_bootstrap.cc:654] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.101317 8528 tablet_bootstrap.cc:492] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Bootstrap starting.
I20260620 23:29:21.101575 8001 catalog_manager.cc:4841] TS 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261): tablet ec8c17be53694446ba1a246d5dc060d8 (table test-table [id=1714876d194b469f98d2777cac05b8f0]) successfully deleted
I20260620 23:29:21.102365 8552 ts_tablet_manager.cc:1905] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:21.102977 8528 tablet_bootstrap.cc:654] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.107560 8553 ts_tablet_manager.cc:1905] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:21.108848 8531 tablet_bootstrap.cc:492] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Bootstrap starting.
I20260620 23:29:21.111588 8531 tablet_bootstrap.cc:654] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.117015 8553 ts_tablet_manager.cc:1918] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:21.117264 8553 log.cc:1196] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/ec8c17be53694446ba1a246d5dc060d8
I20260620 23:29:21.117882 8553 ts_tablet_manager.cc:1939] T ec8c17be53694446ba1a246d5dc060d8 P bc295c278d844c558890837e59b3cf2d: Deleting consensus metadata
I20260620 23:29:21.120285 8001 catalog_manager.cc:4841] TS bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855): tablet ec8c17be53694446ba1a246d5dc060d8 (table test-table [id=1714876d194b469f98d2777cac05b8f0]) successfully deleted
I20260620 23:29:21.126065 8552 ts_tablet_manager.cc:1918] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:21.126295 8552 log.cc:1196] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/ec8c17be53694446ba1a246d5dc060d8
I20260620 23:29:21.126961 8552 ts_tablet_manager.cc:1939] T ec8c17be53694446ba1a246d5dc060d8 P b6275037127246ab967a45bb3c0bad45: Deleting consensus metadata
I20260620 23:29:21.129573 8003 catalog_manager.cc:4841] TS b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535): tablet ec8c17be53694446ba1a246d5dc060d8 (table test-table [id=1714876d194b469f98d2777cac05b8f0]) successfully deleted
I20260620 23:29:21.139045 8531 tablet_bootstrap.cc:492] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: No bootstrap required, opened a new log
I20260620 23:29:21.139289 8531 ts_tablet_manager.cc:1397] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20260620 23:29:21.139789 8528 tablet_bootstrap.cc:492] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: No bootstrap required, opened a new log
I20260620 23:29:21.140009 8528 ts_tablet_manager.cc:1397] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent bootstrapping tablet: real 0.039s user 0.022s sys 0.014s
I20260620 23:29:21.140246 8531 raft_consensus.cc:348] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.140556 8531 raft_consensus.cc:374] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.140678 8531 raft_consensus.cc:729] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc295c278d844c558890837e59b3cf2d, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.141057 8531 consensus_queue.cc:260] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.141043 8528 raft_consensus.cc:348] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.141304 8528 raft_consensus.cc:374] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.141415 8528 raft_consensus.cc:729] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.141752 8528 consensus_queue.cc:260] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.142416 8528 ts_tablet_manager.cc:1428] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:29:21.144706 8531 ts_tablet_manager.cc:1428] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260620 23:29:21.145718 8529 tablet_bootstrap.cc:492] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: No bootstrap required, opened a new log
I20260620 23:29:21.145931 8529 ts_tablet_manager.cc:1397] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.000s
I20260620 23:29:21.146886 8529 raft_consensus.cc:348] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.147218 8529 raft_consensus.cc:374] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.147360 8529 raft_consensus.cc:729] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6275037127246ab967a45bb3c0bad45, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.147657 8529 consensus_queue.cc:260] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.148432 8529 ts_tablet_manager.cc:1428] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:29:21.263545 8534 raft_consensus.cc:482] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:21.263878 8534 raft_consensus.cc:504] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.266117 8534 leader_election.cc:288] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:21.274796 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.266397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38855 (local address 127.0.96.195:45397)
0620 23:29:21.267781 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:21.267799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:21.268023 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:21.268559 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:21.268567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:21.268593 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:21.268911 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.268922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.270510 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.270515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.272404 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.272413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.273877 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.273883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.274084 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.274134 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:29:21.274242 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:29:21.274389 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1278,"mutex_wait_us":148,"thread_start_us":612,"threads_started":1}
I20260620 23:29:21.274868 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.266902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:45397 (local address 127.0.96.193:38855)
0620 23:29:21.267288 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.267294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.267909 (+ 615us) server_negotiation.cc:407] Connection header received
0620 23:29:21.268288 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.268295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.268404 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.268512 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:21.269120 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.270322 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.272610 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.273695 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.273886 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.273964 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.274311 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.274500 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260620 23:29:21.275424 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.266857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:48307 (local address 127.0.96.194:40535)
0620 23:29:21.267149 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.267160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.267601 (+ 441us) server_negotiation.cc:407] Connection header received
0620 23:29:21.267961 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.267969 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.268049 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.268151 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:21.268594 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.269692 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.272251 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.273816 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.274013 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.274099 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.274834 (+ 735us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.275038 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 23:29:21.276170 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.266397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:40535 (local address 127.0.96.195:48307)
0620 23:29:21.267435 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:21.267462 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:21.267753 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:21.268210 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:21.268215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:21.268238 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:21.268418 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.268426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.269819 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.269827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.272027 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.272046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.274145 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.274151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.274317 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.274345 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:29:21.275598 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0620 23:29:21.275808 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":320,"thread_start_us":1111,"threads_started":1}
I20260620 23:29:21.276013 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1bea07282f4e4bbd1b6538a1abc3b4" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d" is_pre_election: true
I20260620 23:29:21.276384 8173 raft_consensus.cc:2454] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:21.276965 8384 leader_election.cc:302] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:21.277205 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1bea07282f4e4bbd1b6538a1abc3b4" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45" is_pre_election: true
I20260620 23:29:21.277523 8534 raft_consensus.cc:2781] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:21.277676 8534 raft_consensus.cc:482] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:21.277787 8534 raft_consensus.cc:3037] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.277769 8314 raft_consensus.cc:2454] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:21.281457 8534 raft_consensus.cc:504] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.282388 8534 leader_election.cc:288] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Requested vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:21.282928 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1bea07282f4e4bbd1b6538a1abc3b4" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45"
I20260620 23:29:21.283020 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed1bea07282f4e4bbd1b6538a1abc3b4" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d"
I20260620 23:29:21.283172 8314 raft_consensus.cc:3037] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.283288 8173 raft_consensus.cc:3037] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.286265 8314 raft_consensus.cc:2454] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:21.286732 8386 leader_election.cc:302] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, b6275037127246ab967a45bb3c0bad45; no voters:
I20260620 23:29:21.286782 8173 raft_consensus.cc:2454] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:21.287117 8534 raft_consensus.cc:2781] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:21.287710 8534 raft_consensus.cc:686] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Becoming Leader. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Running, Role: LEADER
I20260620 23:29:21.288192 8534 consensus_queue.cc:237] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.291702 8016 catalog_manager.cc:5495] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } health_report { overall_health: HEALTHY } } }
Jun 20 23:29:21 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998158, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.254@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
W20260620 23:29:21.371731 8365 tablet.cc:2369] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:21.377336 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.359890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.0.96.254:39409)
0620 23:29:21.360266 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.360276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.360898 (+ 622us) server_negotiation.cc:407] Connection header received
0620 23:29:21.361170 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.361175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.361247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.361349 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:21.363195 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.363930 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.366079 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.366522 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.366643 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.371312 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:21.371317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:21.371320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:21.371350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:21.374563 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:21.375300 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:21.375306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:21.375309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:21.375417 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:21.375791 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:21.375795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:21.375798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:21.376159 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:21.376322 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.376705 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.376919 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260620 23:29:21.377485 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.359780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39409 (local address 127.0.0.1:50732)
0620 23:29:21.360704 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:21.360731 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:21.360968 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:21.361452 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:21.361457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:21.362602 (+ 1145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:21.363002 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.363015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.364108 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.364115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.365875 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.365894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.366770 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.366777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.366958 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.367902 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:21.367935 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:21.371088 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:21.374730 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:21.374747 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:21.374756 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:21.375174 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:21.375534 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:21.375539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:21.375543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:21.375683 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:21.376299 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:21.376307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:21.376467 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 23:29:21.376622 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 23:29:21.377097 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":125,"thread_start_us":517,"threads_started":1}
I20260620 23:29:21.383179 8016 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:50732:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 23:29:21.384253 8016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:21.391897 8293 tablet_service.cc:1467] Processing CreateTablet for tablet ad86447c64914bd3b7023490200bc536 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0f49841be69c44bfa29e56eae173a343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:29:21.391921 8153 tablet_service.cc:1467] Processing CreateTablet for tablet ad86447c64914bd3b7023490200bc536 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0f49841be69c44bfa29e56eae173a343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:29:21.392429 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad86447c64914bd3b7023490200bc536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.392594 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad86447c64914bd3b7023490200bc536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.397919 8529 tablet_bootstrap.cc:492] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: Bootstrap starting.
I20260620 23:29:21.399334 8434 tablet_service.cc:1467] Processing CreateTablet for tablet ad86447c64914bd3b7023490200bc536 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0f49841be69c44bfa29e56eae173a343]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 23:29:21.399727 8529 tablet_bootstrap.cc:654] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.400023 8434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad86447c64914bd3b7023490200bc536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.402926 8529 tablet_bootstrap.cc:492] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: No bootstrap required, opened a new log
I20260620 23:29:21.403106 8529 ts_tablet_manager.cc:1397] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 23:29:21.403733 8529 raft_consensus.cc:348] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.403980 8529 raft_consensus.cc:374] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.404070 8529 raft_consensus.cc:729] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6275037127246ab967a45bb3c0bad45, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.404388 8529 consensus_queue.cc:260] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.404877 8535 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.405705 8535 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.406183 8529 ts_tablet_manager.cc:1428] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 23:29:21.408448 8531 tablet_bootstrap.cc:492] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: Bootstrap starting.
I20260620 23:29:21.410055 8528 tablet_bootstrap.cc:492] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: Bootstrap starting.
I20260620 23:29:21.410646 8531 tablet_bootstrap.cc:654] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.411562 8528 tablet_bootstrap.cc:654] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.413337 8531 tablet_bootstrap.cc:492] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: No bootstrap required, opened a new log
I20260620 23:29:21.413527 8531 ts_tablet_manager.cc:1397] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260620 23:29:21.414214 8528 tablet_bootstrap.cc:492] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: No bootstrap required, opened a new log
I20260620 23:29:21.414395 8528 ts_tablet_manager.cc:1397] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 23:29:21.414355 8531 raft_consensus.cc:348] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.414634 8531 raft_consensus.cc:374] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.414716 8531 raft_consensus.cc:729] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc295c278d844c558890837e59b3cf2d, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.415025 8531 consensus_queue.cc:260] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.415165 8528 raft_consensus.cc:348] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.415468 8528 raft_consensus.cc:374] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.415570 8528 raft_consensus.cc:729] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.415563 8536 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.415824 8528 consensus_queue.cc:260] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.416007 8536 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.416246 8534 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.416577 8534 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.416404 8528 ts_tablet_manager.cc:1428] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 23:29:21.417459 8531 ts_tablet_manager.cc:1428] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260620 23:29:21.444340 8534 raft_consensus.cc:482] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:21.444590 8534 raft_consensus.cc:504] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.445333 8534 leader_election.cc:288] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:21.445958 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad86447c64914bd3b7023490200bc536" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d" is_pre_election: true
I20260620 23:29:21.446058 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad86447c64914bd3b7023490200bc536" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45" is_pre_election: true
I20260620 23:29:21.446237 8173 raft_consensus.cc:2454] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:21.446488 8314 raft_consensus.cc:2454] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:21.446651 8384 leader_election.cc:302] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:21.446975 8534 raft_consensus.cc:2781] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:21.447086 8534 raft_consensus.cc:482] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:21.447198 8534 raft_consensus.cc:3037] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.449407 8534 raft_consensus.cc:504] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.450011 8534 leader_election.cc:288] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Requested vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:21.450350 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad86447c64914bd3b7023490200bc536" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d"
I20260620 23:29:21.450587 8173 raft_consensus.cc:3037] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.450719 8314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad86447c64914bd3b7023490200bc536" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45"
I20260620 23:29:21.451071 8314 raft_consensus.cc:3037] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.452716 8173 raft_consensus.cc:2454] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:21.453197 8384 leader_election.cc:302] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:21.453521 8534 raft_consensus.cc:2781] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:21.453933 8534 raft_consensus.cc:686] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Becoming Leader. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Running, Role: LEADER
I20260620 23:29:21.454268 8534 consensus_queue.cc:237] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.454761 8314 raft_consensus.cc:2454] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:21.456005 8559 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: TxnStatusTablet state changed. Reason: New leader 674db3bc67e54fa7ad72b667bbce7da3. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.456382 8559 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:29:21.457131 8571 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 23:29:21.457368 8571 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 23:29:21.458464 8015 catalog_manager.cc:5495] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } health_report { overall_health: HEALTHY } } }
I20260620 23:29:21.486686 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.473956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:36261 (local address 127.0.0.1:41750)
0620 23:29:21.474342 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:21.474368 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:21.474545 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:21.481898 (+ 7353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:21.481903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:21.481916 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:29:21.482116 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.482125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.483268 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.483276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.484262 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.484273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.485034 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.485039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.485219 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.485280 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:29:21.486147 (+ 867us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:29:21.486167 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 23:29:21.486240 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 23:29:21.486382 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260620 23:29:21.486744 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.474038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.96.195:36261)
0620 23:29:21.481504 (+ 7466us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.481508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.481525 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:29:21.481627 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.481631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.481706 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.481877 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:21.482253 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.483138 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.484407 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.484849 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.484980 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.485475 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:21.486034 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:21.486198 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.486259 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.486345 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7359,"thread_start_us":7147,"threads_started":1}
I20260620 23:29:21.490761 8173 raft_consensus.cc:1261] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:21.490999 8314 raft_consensus.cc:1261] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:21.491536 8559 consensus_queue.cc:1035] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:21.491869 8534 consensus_queue.cc:1035] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:21.502439 8536 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: TxnStatusTablet state changed. Reason: New leader 674db3bc67e54fa7ad72b667bbce7da3. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.502936 8536 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.506101 8559 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.506362 8559 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:29:21.506940 8543 mvcc.cc:205] Tried to move back new op lower bound from 7299064469459009536 to 7299064469319127040. Current Snapshot: MvccSnapshot[applied={T|T < 7299064469459009536 or (T in {7299064469459009536})}]
I20260620 23:29:21.509331 8536 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.509568 8536 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P bc295c278d844c558890837e59b3cf2d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.512779 8535 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: TxnStatusTablet state changed. Reason: New leader 674db3bc67e54fa7ad72b667bbce7da3. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.513109 8535 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.514062 8534 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.514348 8534 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P 674db3bc67e54fa7ad72b667bbce7da3: This TxnStatusTablet replica's current role is: LEADER
I20260620 23:29:21.516052 8535 tablet_replica.cc:439] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } } }
I20260620 23:29:21.516324 8535 tablet_replica.cc:442] T ad86447c64914bd3b7023490200bc536 P b6275037127246ab967a45bb3c0bad45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 23:29:21.526767 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.519873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.0.96.195:36261)
0620 23:29:21.520054 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.520060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.521635 (+ 1575us) server_negotiation.cc:407] Connection header received
0620 23:29:21.521832 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.521838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.521972 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.522058 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:21.522696 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.523454 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.524849 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.525408 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.525484 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.525959 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:21.526068 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:21.526181 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.526343 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.526420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 23:29:21.576849 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.556403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38813 (local address 127.0.96.195:36261)
0620 23:29:21.556660 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.556669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.569889 (+ 13220us) server_negotiation.cc:407] Connection header received
0620 23:29:21.570160 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.570166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.570289 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.570430 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:21.571192 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.572332 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.574492 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.575166 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.575253 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.575870 (+ 617us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:21.576016 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:21.576069 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.576230 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.576307 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:29:21.577796 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.555946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:36261 (local address 127.0.96.195:38813)
0620 23:29:21.569670 (+ 13724us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:21.569702 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:21.569965 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:21.570607 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:21.570613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:21.570625 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 23:29:21.570973 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.570985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.572509 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.572514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.574350 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:21.574363 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.575428 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.575434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:21.575617 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.575694 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 23:29:21.576123 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 23:29:21.576139 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:29:21.577409 (+ 1270us) client_negotiation.cc:240] Negotiation successful
0620 23:29:21.577522 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13437,"thread_start_us":13238,"threads_started":1}
I20260620 23:29:21.592504 8173 raft_consensus.cc:1261] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:21.592762 8314 raft_consensus.cc:1261] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:21.593425 8559 consensus_queue.cc:1035] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:21.593924 8559 consensus_queue.cc:1035] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:21.686403 8017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:21.686681 8017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:21.690472 8017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Sending DeleteTablet for 3 replicas of tablet ed1bea07282f4e4bbd1b6538a1abc3b4
I20260620 23:29:21.691716 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1bea07282f4e4bbd1b6538a1abc3b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:21 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42236
I20260620 23:29:21.691756 8434 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1bea07282f4e4bbd1b6538a1abc3b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:21 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:41738
I20260620 23:29:21.692426 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet ed1bea07282f4e4bbd1b6538a1abc3b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:21 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:33886
I20260620 23:29:21.703156 8588 tablet_replica.cc:330] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: stopping tablet replica
I20260620 23:29:21.703526 8588 raft_consensus.cc:2229] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:21.704216 8588 raft_consensus.cc:2258] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:21.705526 8588 ts_tablet_manager.cc:1905] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:21.707190 8587 tablet_replica.cc:330] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: stopping tablet replica
I20260620 23:29:21.708369 8589 tablet_replica.cc:330] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: stopping tablet replica
I20260620 23:29:21.708751 8589 raft_consensus.cc:2229] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:21.709115 8587 raft_consensus.cc:2229] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:21.709501 8589 raft_consensus.cc:2258] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:21.709872 8587 raft_consensus.cc:2258] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:21.711432 8589 ts_tablet_manager.cc:1905] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:21.712090 8587 ts_tablet_manager.cc:1905] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:21.714854 8588 ts_tablet_manager.cc:1918] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:29:21.715101 8588 log.cc:1196] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/ed1bea07282f4e4bbd1b6538a1abc3b4
I20260620 23:29:21.715723 8588 ts_tablet_manager.cc:1939] T ed1bea07282f4e4bbd1b6538a1abc3b4 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting consensus metadata
I20260620 23:29:21.717149 8001 catalog_manager.cc:4841] TS 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261): tablet ed1bea07282f4e4bbd1b6538a1abc3b4 (table test-table [id=2507b81564094564adde650d2e0b3420]) successfully deleted
I20260620 23:29:21.720851 8587 ts_tablet_manager.cc:1918] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:29:21.721084 8587 log.cc:1196] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/ed1bea07282f4e4bbd1b6538a1abc3b4
I20260620 23:29:21.721660 8587 ts_tablet_manager.cc:1939] T ed1bea07282f4e4bbd1b6538a1abc3b4 P b6275037127246ab967a45bb3c0bad45: Deleting consensus metadata
I20260620 23:29:21.723069 8003 catalog_manager.cc:4841] TS b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535): tablet ed1bea07282f4e4bbd1b6538a1abc3b4 (table test-table [id=2507b81564094564adde650d2e0b3420]) successfully deleted
I20260620 23:29:21.723423 8589 ts_tablet_manager.cc:1918] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:29:21.723585 8589 log.cc:1196] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/ed1bea07282f4e4bbd1b6538a1abc3b4
I20260620 23:29:21.724015 8589 ts_tablet_manager.cc:1939] T ed1bea07282f4e4bbd1b6538a1abc3b4 P bc295c278d844c558890837e59b3cf2d: Deleting consensus metadata
I20260620 23:29:21.725180 8001 catalog_manager.cc:4841] TS bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855): tablet ed1bea07282f4e4bbd1b6538a1abc3b4 (table test-table [id=2507b81564094564adde650d2e0b3420]) successfully deleted
Jun 20 23:29:21 dist-test-slave-v34r krb5kdc[7972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 23:29:21 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:21 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:29:21.778192 8392 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:21.760677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.96.254:39409)
0620 23:29:21.760911 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.760921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.761352 (+ 431us) server_negotiation.cc:407] Connection header received
0620 23:29:21.761602 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.761607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.761695 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.761819 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:21.763270 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.764112 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.765515 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.765976 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.766129 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.777919 (+ 11790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50734: BlockingRecv error: recv got EOF from 127.0.0.1:50734 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 23:29:21 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM
I20260620 23:29:21.799463 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:21.782379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.0.96.254:39409)
0620 23:29:21.782576 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:21.782582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:21.782607 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:29:21.782786 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:21.782795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:21.782887 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:21.782953 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:21.784361 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.785338 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.786748 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:21.787212 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:21.787361 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:21.791939 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:21.791946 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:21.791951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:21.792006 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:21.795828 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:21.796846 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:21.796853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:21.796857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:21.797035 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:21.797507 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:21.797513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:21.797516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:21.797893 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:21.798163 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:21.798755 (+ 592us) server_negotiation.cc:299] Negotiation successful
0620 23:29:21.799035 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:29:21.805411 8017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:29:21.806257 8017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:21.815235 8293 tablet_service.cc:1467] Processing CreateTablet for tablet f95d12c347eb4a54b8d3fed064f7c665 (DEFAULT_TABLE table=test-table [id=850de8117d4c4c339a8776b2e62159f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:21.815249 8153 tablet_service.cc:1467] Processing CreateTablet for tablet f95d12c347eb4a54b8d3fed064f7c665 (DEFAULT_TABLE table=test-table [id=850de8117d4c4c339a8776b2e62159f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:21.815441 8434 tablet_service.cc:1467] Processing CreateTablet for tablet f95d12c347eb4a54b8d3fed064f7c665 (DEFAULT_TABLE table=test-table [id=850de8117d4c4c339a8776b2e62159f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:21.815724 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95d12c347eb4a54b8d3fed064f7c665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.815826 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95d12c347eb4a54b8d3fed064f7c665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.815840 8434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95d12c347eb4a54b8d3fed064f7c665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:21.822395 8528 tablet_bootstrap.cc:492] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Bootstrap starting.
I20260620 23:29:21.822705 8529 tablet_bootstrap.cc:492] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Bootstrap starting.
I20260620 23:29:21.823226 8531 tablet_bootstrap.cc:492] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Bootstrap starting.
I20260620 23:29:21.824211 8529 tablet_bootstrap.cc:654] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.824226 8528 tablet_bootstrap.cc:654] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.825232 8531 tablet_bootstrap.cc:654] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:21.835436 8529 tablet_bootstrap.cc:492] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: No bootstrap required, opened a new log
I20260620 23:29:21.835726 8529 ts_tablet_manager.cc:1397] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260620 23:29:21.836582 8529 raft_consensus.cc:348] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.837011 8529 raft_consensus.cc:374] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.837148 8529 raft_consensus.cc:729] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6275037127246ab967a45bb3c0bad45, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.837440 8529 consensus_queue.cc:260] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.838541 8531 tablet_bootstrap.cc:492] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: No bootstrap required, opened a new log
I20260620 23:29:21.838853 8531 ts_tablet_manager.cc:1397] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20260620 23:29:21.839916 8531 raft_consensus.cc:348] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.840215 8531 raft_consensus.cc:374] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.840329 8531 raft_consensus.cc:729] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc295c278d844c558890837e59b3cf2d, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.840360 8529 ts_tablet_manager.cc:1428] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260620 23:29:21.840643 8531 consensus_queue.cc:260] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.841440 8531 ts_tablet_manager.cc:1428] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 23:29:21.842499 8528 tablet_bootstrap.cc:492] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: No bootstrap required, opened a new log
I20260620 23:29:21.842666 8528 ts_tablet_manager.cc:1397] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.007s
I20260620 23:29:21.843726 8528 raft_consensus.cc:348] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.844031 8528 raft_consensus.cc:374] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:21.844133 8528 raft_consensus.cc:729] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Initialized, Role: FOLLOWER
I20260620 23:29:21.844373 8528 consensus_queue.cc:260] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.845944 8528 ts_tablet_manager.cc:1428] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 23:29:21.975471 8585 raft_consensus.cc:482] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:21.975683 8585 raft_consensus.cc:504] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.976624 8585 leader_election.cc:288] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:21.977401 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95d12c347eb4a54b8d3fed064f7c665" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d" is_pre_election: true
I20260620 23:29:21.977722 8173 raft_consensus.cc:2454] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:21.977711 8313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95d12c347eb4a54b8d3fed064f7c665" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45" is_pre_election: true
I20260620 23:29:21.978242 8313 raft_consensus.cc:2454] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:21.978300 8384 leader_election.cc:302] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:21.978674 8585 raft_consensus.cc:2781] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:21.978829 8585 raft_consensus.cc:482] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:21.978991 8585 raft_consensus.cc:3037] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.981463 8585 raft_consensus.cc:504] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.982182 8585 leader_election.cc:288] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Requested vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:21.982638 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95d12c347eb4a54b8d3fed064f7c665" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d"
I20260620 23:29:21.982848 8173 raft_consensus.cc:3037] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.983011 8313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95d12c347eb4a54b8d3fed064f7c665" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45"
I20260620 23:29:21.983347 8313 raft_consensus.cc:3037] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:21.985235 8173 raft_consensus.cc:2454] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:21.985774 8384 leader_election.cc:302] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:21.986090 8585 raft_consensus.cc:2781] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:21.986254 8585 raft_consensus.cc:686] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Becoming Leader. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Running, Role: LEADER
I20260620 23:29:21.986582 8585 consensus_queue.cc:237] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:21.986792 8313 raft_consensus.cc:2454] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:21.989863 8017 catalog_manager.cc:5495] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } health_report { overall_health: HEALTHY } } }
I20260620 23:29:22.031075 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:22.022344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.96.195:36261)
0620 23:29:22.022628 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:22.022634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:22.023900 (+ 1266us) server_negotiation.cc:407] Connection header received
0620 23:29:22.024010 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:22.024017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:22.024137 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:22.024242 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:22.024969 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.026228 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.028441 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.029397 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.029560 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:22.030212 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:22.030343 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:22.030438 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:22.030627 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 23:29:22.030698 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 23:29:22.033977 8173 raft_consensus.cc:1261] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:22.034140 8313 raft_consensus.cc:1261] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:22.034696 8585 consensus_queue.cc:1035] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:22.035120 8582 consensus_queue.cc:1035] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:22.050716 8016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:22.051011 8016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:22.054028 8016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Sending DeleteTablet for 3 replicas of tablet f95d12c347eb4a54b8d3fed064f7c665
I20260620 23:29:22.054914 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet f95d12c347eb4a54b8d3fed064f7c665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:22 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:33886
I20260620 23:29:22.054953 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet f95d12c347eb4a54b8d3fed064f7c665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:22 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42236
I20260620 23:29:22.055361 8589 tablet_replica.cc:330] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: stopping tablet replica
I20260620 23:29:22.055445 8434 tablet_service.cc:1514] Processing DeleteTablet for tablet f95d12c347eb4a54b8d3fed064f7c665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:22 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:41738
I20260620 23:29:22.055642 8589 raft_consensus.cc:2229] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:22.055756 8588 tablet_replica.cc:330] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: stopping tablet replica
I20260620 23:29:22.055928 8589 raft_consensus.cc:2258] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:22.055967 8588 raft_consensus.cc:2229] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:22.056277 8587 tablet_replica.cc:330] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: stopping tablet replica
I20260620 23:29:22.056283 8588 raft_consensus.cc:2258] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:22.056492 8587 raft_consensus.cc:2229] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:22.056769 8587 raft_consensus.cc:2258] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:22.057507 8588 ts_tablet_manager.cc:1905] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:22.057898 8587 ts_tablet_manager.cc:1905] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:22.058995 8589 ts_tablet_manager.cc:1905] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:22.064500 8587 ts_tablet_manager.cc:1918] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:22.064551 8588 ts_tablet_manager.cc:1918] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:22.064679 8587 log.cc:1196] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/f95d12c347eb4a54b8d3fed064f7c665
I20260620 23:29:22.064687 8588 log.cc:1196] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/f95d12c347eb4a54b8d3fed064f7c665
I20260620 23:29:22.065135 8587 ts_tablet_manager.cc:1939] T f95d12c347eb4a54b8d3fed064f7c665 P b6275037127246ab967a45bb3c0bad45: Deleting consensus metadata
I20260620 23:29:22.065153 8588 ts_tablet_manager.cc:1939] T f95d12c347eb4a54b8d3fed064f7c665 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting consensus metadata
I20260620 23:29:22.066160 8016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 23:29:22.066708 8589 ts_tablet_manager.cc:1918] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:22.066859 8589 log.cc:1196] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/f95d12c347eb4a54b8d3fed064f7c665
W20260620 23:29:22.066849 8016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:22.067184 8589 ts_tablet_manager.cc:1939] T f95d12c347eb4a54b8d3fed064f7c665 P bc295c278d844c558890837e59b3cf2d: Deleting consensus metadata
I20260620 23:29:22.068498 8003 catalog_manager.cc:4841] TS b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535): tablet f95d12c347eb4a54b8d3fed064f7c665 (table test-table [id=850de8117d4c4c339a8776b2e62159f2]) successfully deleted
I20260620 23:29:22.068725 8001 catalog_manager.cc:4841] TS bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855): tablet f95d12c347eb4a54b8d3fed064f7c665 (table test-table [id=850de8117d4c4c339a8776b2e62159f2]) successfully deleted
I20260620 23:29:22.069662 8001 catalog_manager.cc:4841] TS 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261): tablet f95d12c347eb4a54b8d3fed064f7c665 (table test-table [id=850de8117d4c4c339a8776b2e62159f2]) successfully deleted
I20260620 23:29:22.073968 8153 tablet_service.cc:1467] Processing CreateTablet for tablet b262806cc58c4520bbc0bbb50d3721d3 (DEFAULT_TABLE table=test-table [id=5e2d5ffb963c4bfe8129d75071468078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:22.074219 8434 tablet_service.cc:1467] Processing CreateTablet for tablet b262806cc58c4520bbc0bbb50d3721d3 (DEFAULT_TABLE table=test-table [id=5e2d5ffb963c4bfe8129d75071468078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:22.074553 8153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b262806cc58c4520bbc0bbb50d3721d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:22.074752 8434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b262806cc58c4520bbc0bbb50d3721d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:22.078408 8293 tablet_service.cc:1467] Processing CreateTablet for tablet b262806cc58c4520bbc0bbb50d3721d3 (DEFAULT_TABLE table=test-table [id=5e2d5ffb963c4bfe8129d75071468078]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:22.078984 8293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b262806cc58c4520bbc0bbb50d3721d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:22.080106 8528 tablet_bootstrap.cc:492] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Bootstrap starting.
I20260620 23:29:22.081676 8528 tablet_bootstrap.cc:654] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:22.086275 8529 tablet_bootstrap.cc:492] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Bootstrap starting.
I20260620 23:29:22.087165 8531 tablet_bootstrap.cc:492] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Bootstrap starting.
I20260620 23:29:22.088536 8529 tablet_bootstrap.cc:654] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:22.088954 8531 tablet_bootstrap.cc:654] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:22.090632 8528 tablet_bootstrap.cc:492] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: No bootstrap required, opened a new log
I20260620 23:29:22.090809 8528 ts_tablet_manager.cc:1397] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 23:29:22.091399 8531 tablet_bootstrap.cc:492] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: No bootstrap required, opened a new log
I20260620 23:29:22.091615 8531 ts_tablet_manager.cc:1397] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 23:29:22.091774 8528 raft_consensus.cc:348] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.092060 8528 raft_consensus.cc:374] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:22.092200 8528 raft_consensus.cc:729] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Initialized, Role: FOLLOWER
I20260620 23:29:22.092571 8531 raft_consensus.cc:348] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.092607 8528 consensus_queue.cc:260] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.092922 8531 raft_consensus.cc:374] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:22.093068 8531 raft_consensus.cc:729] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc295c278d844c558890837e59b3cf2d, State: Initialized, Role: FOLLOWER
I20260620 23:29:22.093410 8531 consensus_queue.cc:260] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.093979 8528 ts_tablet_manager.cc:1428] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 23:29:22.095201 8529 tablet_bootstrap.cc:492] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: No bootstrap required, opened a new log
I20260620 23:29:22.095420 8529 ts_tablet_manager.cc:1397] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260620 23:29:22.096318 8531 ts_tablet_manager.cc:1428] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260620 23:29:22.096998 8529 raft_consensus.cc:348] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.097295 8529 raft_consensus.cc:374] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:22.097419 8529 raft_consensus.cc:729] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6275037127246ab967a45bb3c0bad45, State: Initialized, Role: FOLLOWER
I20260620 23:29:22.097728 8529 consensus_queue.cc:260] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.098278 8529 ts_tablet_manager.cc:1428] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 23:29:22.163972 8585 raft_consensus.cc:482] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:22.164283 8585 raft_consensus.cc:504] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.165230 8585 leader_election.cc:288] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:22.165946 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b262806cc58c4520bbc0bbb50d3721d3" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d" is_pre_election: true
I20260620 23:29:22.166090 8313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b262806cc58c4520bbc0bbb50d3721d3" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45" is_pre_election: true
I20260620 23:29:22.166270 8173 raft_consensus.cc:2454] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:22.166404 8313 raft_consensus.cc:2454] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 0.
I20260620 23:29:22.166918 8386 leader_election.cc:302] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, b6275037127246ab967a45bb3c0bad45, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:22.167330 8585 raft_consensus.cc:2781] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:22.167472 8585 raft_consensus.cc:482] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:22.167586 8585 raft_consensus.cc:3037] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:22.171145 8585 raft_consensus.cc:504] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.172255 8585 leader_election.cc:288] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Requested vote from peers bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855), b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535)
I20260620 23:29:22.172730 8173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b262806cc58c4520bbc0bbb50d3721d3" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc295c278d844c558890837e59b3cf2d"
I20260620 23:29:22.172830 8313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b262806cc58c4520bbc0bbb50d3721d3" candidate_uuid: "674db3bc67e54fa7ad72b667bbce7da3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6275037127246ab967a45bb3c0bad45"
I20260620 23:29:22.172961 8173 raft_consensus.cc:3037] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:22.173115 8313 raft_consensus.cc:3037] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:22.175256 8173 raft_consensus.cc:2454] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:22.175727 8384 leader_election.cc:302] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 674db3bc67e54fa7ad72b667bbce7da3, bc295c278d844c558890837e59b3cf2d; no voters:
I20260620 23:29:22.176128 8585 raft_consensus.cc:2781] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:22.176313 8585 raft_consensus.cc:686] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Becoming Leader. State: Replica: 674db3bc67e54fa7ad72b667bbce7da3, State: Running, Role: LEADER
I20260620 23:29:22.176512 8313 raft_consensus.cc:2454] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 674db3bc67e54fa7ad72b667bbce7da3 in term 1.
I20260620 23:29:22.176630 8585 consensus_queue.cc:237] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } }
I20260620 23:29:22.180451 8017 catalog_manager.cc:5495] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "674db3bc67e54fa7ad72b667bbce7da3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "674db3bc67e54fa7ad72b667bbce7da3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36261 } health_report { overall_health: HEALTHY } } }
I20260620 23:29:22.219069 8173 raft_consensus.cc:1261] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:22.219573 8313 raft_consensus.cc:1261] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Refusing update from remote peer 674db3bc67e54fa7ad72b667bbce7da3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:22.219709 8559 consensus_queue.cc:1035] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc295c278d844c558890837e59b3cf2d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:22.220237 8585 consensus_queue.cc:1035] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6275037127246ab967a45bb3c0bad45" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 40535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:22.287830 8017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:22.288079 8017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:22.291827 8017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37ca96b109df42008436fa0d06c69111: Sending DeleteTablet for 3 replicas of tablet b262806cc58c4520bbc0bbb50d3721d3
I20260620 23:29:22.292912 8153 tablet_service.cc:1514] Processing DeleteTablet for tablet b262806cc58c4520bbc0bbb50d3721d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:22 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:33886
I20260620 23:29:22.292934 8293 tablet_service.cc:1514] Processing DeleteTablet for tablet b262806cc58c4520bbc0bbb50d3721d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:22 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42236
I20260620 23:29:22.293288 8587 tablet_replica.cc:330] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: stopping tablet replica
I20260620 23:29:22.293311 8589 tablet_replica.cc:330] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: stopping tablet replica
I20260620 23:29:22.293480 8589 raft_consensus.cc:2229] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:22.293490 8587 raft_consensus.cc:2229] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:22.293720 8589 raft_consensus.cc:2258] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:22.293763 8587 raft_consensus.cc:2258] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:22.294061 8434 tablet_service.cc:1514] Processing DeleteTablet for tablet b262806cc58c4520bbc0bbb50d3721d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:22 UTC) from {username='oryx', principal='oryx/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:41738
I20260620 23:29:22.294559 8588 tablet_replica.cc:330] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: stopping tablet replica
I20260620 23:29:22.294807 8588 raft_consensus.cc:2229] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:22.295087 8587 ts_tablet_manager.cc:1905] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:22.295094 8589 ts_tablet_manager.cc:1905] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:22.295224 8588 raft_consensus.cc:2258] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:22.296622 8588 ts_tablet_manager.cc:1905] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:22.299759 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 8088
I20260620 23:29:22.302186 8589 ts_tablet_manager.cc:1918] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:29:22.302472 8589 log.cc:1196] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/b262806cc58c4520bbc0bbb50d3721d3
I20260620 23:29:22.303031 8589 ts_tablet_manager.cc:1939] T b262806cc58c4520bbc0bbb50d3721d3 P bc295c278d844c558890837e59b3cf2d: Deleting consensus metadata
I20260620 23:29:22.305089 8001 catalog_manager.cc:4841] TS bc295c278d844c558890837e59b3cf2d (127.0.96.193:38855): tablet b262806cc58c4520bbc0bbb50d3721d3 (table test-table [id=5e2d5ffb963c4bfe8129d75071468078]) successfully deleted
I20260620 23:29:22.305460 8588 ts_tablet_manager.cc:1918] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:29:22.305776 8588 log.cc:1196] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/b262806cc58c4520bbc0bbb50d3721d3
I20260620 23:29:22.306289 8588 ts_tablet_manager.cc:1939] T b262806cc58c4520bbc0bbb50d3721d3 P 674db3bc67e54fa7ad72b667bbce7da3: Deleting consensus metadata
I20260620 23:29:22.307619 8001 catalog_manager.cc:4841] TS 674db3bc67e54fa7ad72b667bbce7da3 (127.0.96.195:36261): tablet b262806cc58c4520bbc0bbb50d3721d3 (table test-table [id=5e2d5ffb963c4bfe8129d75071468078]) successfully deleted
I20260620 23:29:22.309844 8587 ts_tablet_manager.cc:1918] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 23:29:22.310007 8587 log.cc:1196] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/b262806cc58c4520bbc0bbb50d3721d3
I20260620 23:29:22.310491 8587 ts_tablet_manager.cc:1939] T b262806cc58c4520bbc0bbb50d3721d3 P b6275037127246ab967a45bb3c0bad45: Deleting consensus metadata
I20260620 23:29:22.312597 8003 catalog_manager.cc:4841] TS b6275037127246ab967a45bb3c0bad45 (127.0.96.194:40535): tablet b262806cc58c4520bbc0bbb50d3721d3 (table test-table [id=5e2d5ffb963c4bfe8129d75071468078]) successfully deleted
Jun 20 23:29:22 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.193@KRBTEST.COM
I20260620 23:29:22.332469 8619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:22.301397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.0.96.193:38855)
0620 23:29:22.304698 (+ 3301us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:22.304704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:22.310215 (+ 5511us) server_negotiation.cc:407] Connection header received
0620 23:29:22.310883 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:22.310890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:22.311000 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:22.311135 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:22.313197 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.314336 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.316640 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.317742 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.317931 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:22.324055 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:22.324086 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:22.324115 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:22.324183 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:22.328778 (+ 4595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:22.329799 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:22.329811 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:22.329814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:22.330016 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:22.330443 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:22.330448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:22.330453 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:22.330999 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:22.331350 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:22.331617 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 23:29:22.332008 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3189,"thread_start_us":3101,"threads_started":1}
I20260620 23:29:22.333312 8219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:22.470271 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 8088
I20260620 23:29:22.490031 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 8228
Jun 20 23:29:22 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.194@KRBTEST.COM
I20260620 23:29:22.508543 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:22.491002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42248 (local address 127.0.96.194:40535)
0620 23:29:22.491929 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:22.491933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:22.491949 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:29:22.492003 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:22.492008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:22.492077 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:22.492560 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:22.493982 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.494715 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.496041 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.496447 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.496599 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:22.501577 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:22.501594 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:22.501613 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:22.501664 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:22.505051 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:22.506168 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:22.506175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:22.506177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:22.506338 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:22.506862 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:22.506869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:22.506874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:22.507293 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:22.507537 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:22.508043 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 23:29:22.508227 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":708,"threads_started":1}
I20260620 23:29:22.509531 8359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:22.635520 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 8228
I20260620 23:29:22.659559 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 8368
Jun 20 23:29:22 dist-test-slave-v34r krb5kdc[7972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.96.195@KRBTEST.COM
I20260620 23:29:22.678143 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:22.660388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41762 (local address 127.0.96.195:36261)
0620 23:29:22.660869 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:22.660873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:22.660892 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:29:22.660963 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:22.660967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:22.661162 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:22.661300 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:22.662985 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.663832 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.665606 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:22.666037 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:22.666170 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:22.671405 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:22.671420 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:22.671437 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:22.671476 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:22.674708 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:22.675663 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:22.675670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:22.675672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:22.675796 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:22.676183 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:22.676186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:22.676188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:22.676592 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:22.676816 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:22.677395 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 23:29:22.677683 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":268,"threads_started":1}
I20260620 23:29:22.678987 8499 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:22.825492 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 8368
I20260620 23:29:22.852465 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 7985
I20260620 23:29:22.854095 8057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:22.971097 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 7985
2026-06-20T23:29:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6097 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:23 dist-test-slave-v34r krb5kdc[8628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:23 dist-test-slave-v34r krb5kdc[8628](info): set up 2 sockets
Jun 20 23:29:23 dist-test-slave-v34r krb5kdc[8628](info): commencing operation
krb5kdc: starting...
W20260620 23:29:24.098170 387 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:24 dist-test-slave-v34r krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:29:24.357641 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35277
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:24.712159 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:24.712509 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:24.712675 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:24.720034 8641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:24.720211 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:24.720319 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:24.720423 8641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:24.720492 8641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:24.730659 8641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:24.732127 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:24.733477 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:24.746634 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:24.746776 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:24.746634 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:24.748628 8641 server_base.cc:1034] running on GCE node
I20260620 23:29:24.749577 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:24.751132 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:24.752417 8641 hybrid_clock.cc:648] HybridClock initialized: now 1781998164752352 us; error 65 us; skew 500 ppm
I20260620 23:29:24.752832 8641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T23:29:24Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1852 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:24 dist-test-slave-v34r krb5kdc[8655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:24 dist-test-slave-v34r krb5kdc[8655](info): set up 2 sockets
Jun 20 23:29:24 dist-test-slave-v34r krb5kdc[8655](info): commencing operation
krb5kdc: starting...
W20260620 23:29:25.966228 387 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:26 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:29:26.241549 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38917
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:26.656064 8668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:26.656570 8668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:26.656759 8668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:26.665043 8668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:26.665273 8668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:26.665375 8668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:26.665498 8668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:26.665592 8668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:26.676865 8668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:26.678705 8668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:26.680387 8668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:26.695482 8673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:26.695456 8674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:26.695394 8676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:26.696863 8668 server_base.cc:1034] running on GCE node
I20260620 23:29:26.697844 8668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:26.699568 8668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:26.700825 8668 hybrid_clock.cc:648] HybridClock initialized: now 1781998166700777 us; error 40 us; skew 500 ppm
I20260620 23:29:26.701301 8668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:26 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:26.705135 8668 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:29:26.707091 8668 webserver.cc:460] Webserver started at http://127.0.96.254:38131/ using document root <none> and password file <none>
I20260620 23:29:26.707973 8668 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:26.708169 8668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:26.708606 8668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:26.712002 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5"
format_stamp: "Formatted at 2026-06-20 23:29:26 on dist-test-slave-v34r"
server_key: "e9864eac835f37a39752bd62cdef2824"
server_key_iv: "c3674d80958085e492a3d22f4fbd49aa"
server_key_version: "encryptionkey@0"
I20260620 23:29:26.712893 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5"
format_stamp: "Formatted at 2026-06-20 23:29:26 on dist-test-slave-v34r"
server_key: "e9864eac835f37a39752bd62cdef2824"
server_key_iv: "c3674d80958085e492a3d22f4fbd49aa"
server_key_version: "encryptionkey@0"
I20260620 23:29:26.719144 8668 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260620 23:29:26.723368 8683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:26.725447 8668 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:29:26.725673 8668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5"
format_stamp: "Formatted at 2026-06-20 23:29:26 on dist-test-slave-v34r"
server_key: "e9864eac835f37a39752bd62cdef2824"
server_key_iv: "c3674d80958085e492a3d22f4fbd49aa"
server_key_version: "encryptionkey@0"
I20260620 23:29:26.725889 8668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:26.758982 8668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:26.763921 8668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:26.764294 8668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:26.785549 8668 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38917
I20260620 23:29:26.785573 8734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38917 every 8 connection(s)
I20260620 23:29:26.787024 8668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:26.790359 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 8668
I20260620 23:29:26.790532 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:26.791293 387 external_mini_cluster.cc:1428] Setting key c3ac6486a9751d89bd789748e7c5020e
I20260620 23:29:26.792811 8735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:26.809653 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Bootstrap starting.
Jun 20 23:29:26 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:26.814252 8735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:26.816076 8735 log.cc:824] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:26.820806 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: No bootstrap required, opened a new log
I20260620 23:29:26.824812 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:26.793789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.0.96.254:38917)
0620 23:29:26.794726 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:26.794739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:26.794783 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:29:26.796496 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:26.796521 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:26.797075 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:26.797469 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:26.799832 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:26.801407 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:26.803493 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:26.804270 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:26.804441 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:26.814063 (+ 9622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:26.814081 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:26.814118 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:26.814191 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:26.818699 (+ 4508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:26.819932 (+ 1233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:26.819942 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:26.819954 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:26.820235 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:26.820710 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:26.820715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:26.820718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:26.821694 (+ 976us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:26.822084 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:26.822498 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 23:29:26.823081 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":422,"threads_started":1}
I20260620 23:29:26.831076 8735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:26.831584 8735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:26.831674 8735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2e8f7b3b0641fcb333c7d4c071e9d5, State: Initialized, Role: FOLLOWER
I20260620 23:29:26.832394 8735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:26.832648 8735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:26.832818 8735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:26.833019 8735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:26.836086 8735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:26.836797 8735 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2e8f7b3b0641fcb333c7d4c071e9d5; no voters:
I20260620 23:29:26.837536 8735 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:26.837751 8740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:26.839329 8740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3c2e8f7b3b0641fcb333c7d4c071e9d5, State: Running, Role: LEADER
I20260620 23:29:26.839964 8740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:26.840543 8735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:26.843545 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2e8f7b3b0641fcb333c7d4c071e9d5. Latest consensus state: current_term: 1 leader_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } } }
I20260620 23:29:26.843906 8741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } } }
I20260620 23:29:26.844201 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:26.844278 8741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:26.846031 8745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:26.852269 8745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:26.863987 8745 catalog_manager.cc:1347] Generated new cluster ID: 97099eb68692441a913b4d46a4849186
I20260620 23:29:26.864194 8745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:26.887449 8745 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:26.889598 8745 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:26.900602 8745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Generated new TSK 0
I20260620 23:29:26.901862 8745 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:29:26.977259 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38917
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:27.406059 8761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:27.406454 8761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:27.406558 8761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:27.415158 8761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:27.415323 8761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:27.415458 8761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:27.427016 8761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:27.428879 8761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:27.430722 8761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:27.447785 8769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:27.447808 8766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:27.447935 8767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:27.448714 8761 server_base.cc:1034] running on GCE node
I20260620 23:29:27.449681 8761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:27.451038 8761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:27.452325 8761 hybrid_clock.cc:648] HybridClock initialized: now 1781998167452263 us; error 56 us; skew 500 ppm
I20260620 23:29:27.453163 8761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:27 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:27.457486 8761 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:29:27.459586 8761 webserver.cc:460] Webserver started at http://127.0.96.193:41303/ using document root <none> and password file <none>
I20260620 23:29:27.460599 8761 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:27.460783 8761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:27.461258 8761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:27.464962 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "9898e145ac22454bba182bd5b47a4c1d"
format_stamp: "Formatted at 2026-06-20 23:29:27 on dist-test-slave-v34r"
server_key: "dd9916547f6cee1491d306813387e81a"
server_key_iv: "47cc3fa82d63f8e58652fc3e265a9e20"
server_key_version: "encryptionkey@0"
I20260620 23:29:27.466107 8761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "9898e145ac22454bba182bd5b47a4c1d"
format_stamp: "Formatted at 2026-06-20 23:29:27 on dist-test-slave-v34r"
server_key: "dd9916547f6cee1491d306813387e81a"
server_key_iv: "47cc3fa82d63f8e58652fc3e265a9e20"
server_key_version: "encryptionkey@0"
I20260620 23:29:27.473125 8761 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260620 23:29:27.477888 8776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:27.480000 8761 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:29:27.480289 8761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "9898e145ac22454bba182bd5b47a4c1d"
format_stamp: "Formatted at 2026-06-20 23:29:27 on dist-test-slave-v34r"
server_key: "dd9916547f6cee1491d306813387e81a"
server_key_iv: "47cc3fa82d63f8e58652fc3e265a9e20"
server_key_version: "encryptionkey@0"
I20260620 23:29:27.480594 8761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:27.510113 8761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:27.515715 8761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:27.516121 8761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:27.517439 8761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:27.519312 8761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:27.519430 8761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:27.519603 8761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:27.519680 8761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:27.559717 8761 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:39957
I20260620 23:29:27.559820 8888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:39957 every 8 connection(s)
I20260620 23:29:27.561344 8761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:27.571411 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 8761
I20260620 23:29:27.571660 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:27.572530 387 external_mini_cluster.cc:1428] Setting key f7b33c7e5546c43ebbf92cab19adc230
Jun 20 23:29:27 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:27.588109 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:27.564825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52071 (local address 127.0.96.254:38917)
0620 23:29:27.565444 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:27.565452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:27.567009 (+ 1557us) server_negotiation.cc:407] Connection header received
0620 23:29:27.569108 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:27.569113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:27.569221 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:27.569366 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:27.571211 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:27.572509 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:27.574534 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:27.575179 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:27.575313 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:27.580312 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:27.580322 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:27.580331 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:27.580388 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:27.583856 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:27.585232 (+ 1376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:27.585238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:27.585241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:27.585421 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:27.585976 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:27.585980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:27.585982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:27.586295 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:27.586476 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:27.587371 (+ 895us) server_negotiation.cc:299] Negotiation successful
0620 23:29:27.587682 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":280,"threads_started":1}
I20260620 23:29:27.589547 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:27.565724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:52071)
0620 23:29:27.566762 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:27.566826 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:27.568705 (+ 1879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:27.569685 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:27.569694 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:27.570299 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:27.570956 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:27.570974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:27.572771 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:27.572776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:27.574328 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:27.574342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:27.575455 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:27.575463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:27.575802 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:27.576703 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:27.576741 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:27.579942 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:27.584127 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:27.584144 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:27.584175 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:27.584959 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:27.585560 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:27.585566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:27.585570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:27.585812 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:27.586443 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:27.586454 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:27.586925 (+ 471us) client_negotiation.cc:769] Sending connection context
0620 23:29:27.587250 (+ 325us) client_negotiation.cc:240] Negotiation successful
0620 23:29:27.587969 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":407,"threads_started":1}
I20260620 23:29:27.591884 8889 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38917
I20260620 23:29:27.592305 8889 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:27.593423 8889 heartbeater.cc:511] Master 127.0.96.254:38917 requested a full tablet report, sending...
I20260620 23:29:27.596518 8700 ts_manager.cc:194] Registered new tserver with Master: 9898e145ac22454bba182bd5b47a4c1d (127.0.96.193:39957)
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:29:27.598562 8700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:52071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:29:27.628845 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38917
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:27.986622 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:27.986901 8895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:27.987053 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:27.994629 8895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:27.994745 8895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:27.994923 8895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:28.004897 8895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:28.006533 8895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:28.008011 8895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:28.024399 8900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:28.024792 8903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:28.025396 8901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:28.028580 8895 server_base.cc:1034] running on GCE node
I20260620 23:29:28.029722 8895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:28.031419 8895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:28.032668 8895 hybrid_clock.cc:648] HybridClock initialized: now 1781998168032606 us; error 49 us; skew 500 ppm
I20260620 23:29:28.033151 8895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:28 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:28.037329 8895 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:29:28.039463 8895 webserver.cc:460] Webserver started at http://127.0.96.194:33737/ using document root <none> and password file <none>
I20260620 23:29:28.040452 8895 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:28.040633 8895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:28.041106 8895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:28.044906 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "1c452d48dd6e4c86805de8407fb7f6c0"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "64223f74874f7cd3fcd0213b2cfb76d9"
server_key_iv: "bc9c4c3f83d7c40dcd39e92681ce4ee3"
server_key_version: "encryptionkey@0"
I20260620 23:29:28.046021 8895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "1c452d48dd6e4c86805de8407fb7f6c0"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "64223f74874f7cd3fcd0213b2cfb76d9"
server_key_iv: "bc9c4c3f83d7c40dcd39e92681ce4ee3"
server_key_version: "encryptionkey@0"
I20260620 23:29:28.053109 8895 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 23:29:28.057924 8910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:28.059801 8895 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:29:28.060067 8895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "1c452d48dd6e4c86805de8407fb7f6c0"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "64223f74874f7cd3fcd0213b2cfb76d9"
server_key_iv: "bc9c4c3f83d7c40dcd39e92681ce4ee3"
server_key_version: "encryptionkey@0"
I20260620 23:29:28.060353 8895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:28.081081 8895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:28.087141 8895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:28.087525 8895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:28.088932 8895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:28.090847 8895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:28.090970 8895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:28.091120 8895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:28.091180 8895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:28.134205 8895 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:44635
I20260620 23:29:28.134495 9022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:44635 every 8 connection(s)
I20260620 23:29:28.136567 8895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:28.143895 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 8895
I20260620 23:29:28.144098 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:28.144649 387 external_mini_cluster.cc:1428] Setting key 4e08155ead6556f9d6fa0b1106d15cf3
Jun 20 23:29:28 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:28.163976 9025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:28.139992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:44915 (local address 127.0.96.254:38917)
0620 23:29:28.140567 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:28.140572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:28.142010 (+ 1438us) server_negotiation.cc:407] Connection header received
0620 23:29:28.144397 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:28.144404 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:28.144499 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:28.144620 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:28.147147 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.148306 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.150362 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.151105 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.151261 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:28.156487 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:28.156493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:28.156498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:28.156559 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:28.159566 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:28.161086 (+ 1520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:28.161094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:28.161097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:28.161241 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:28.161889 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:28.161892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:28.161894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:28.162186 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:28.162307 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:28.163511 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0620 23:29:28.163711 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":302,"threads_started":1}
I20260620 23:29:28.166114 9026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:28.140770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:44915)
0620 23:29:28.141829 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:28.141876 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:28.143916 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:28.145099 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:28.145111 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:28.145974 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:28.146912 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:28.146940 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.148515 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.148520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:28.150169 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:28.150186 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.151413 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.151419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:28.151694 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:28.152612 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:28.152656 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:28.156059 (+ 3403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:28.159897 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:28.159914 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:28.159944 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:28.160790 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:28.161394 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:28.161402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:28.161408 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:28.161738 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:28.162331 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:28.162343 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:28.162995 (+ 652us) client_negotiation.cc:769] Sending connection context
0620 23:29:28.163376 (+ 381us) client_negotiation.cc:240] Negotiation successful
0620 23:29:28.164134 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":449,"threads_started":1}
I20260620 23:29:28.167953 9023 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38917
I20260620 23:29:28.168331 9023 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:28.169210 9023 heartbeater.cc:511] Master 127.0.96.254:38917 requested a full tablet report, sending...
I20260620 23:29:28.170810 8700 ts_manager.cc:194] Registered new tserver with Master: 1c452d48dd6e4c86805de8407fb7f6c0 (127.0.96.194:44635)
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:29:28.171730 8700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:44915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:29:28.202694 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38917
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:28.602797 8889 heartbeater.cc:503] Master 127.0.96.254:38917 was elected leader, sending a full tablet report...
W20260620 23:29:28.613190 9029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:28.613674 9029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:28.613873 9029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:28.622706 9029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:28.622916 9029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:28.623121 9029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:28.635602 9029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:28.637509 9029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:28.639614 9029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:28.656288 9035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:28.656323 9034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:28.656373 9037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:28.657775 9029 server_base.cc:1034] running on GCE node
I20260620 23:29:28.658761 9029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:28.660053 9029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:28.661312 9029 hybrid_clock.cc:648] HybridClock initialized: now 1781998168661261 us; error 45 us; skew 500 ppm
I20260620 23:29:28.661733 9029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:28 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:28.666525 9029 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:29:28.668844 9029 webserver.cc:460] Webserver started at http://127.0.96.195:36507/ using document root <none> and password file <none>
I20260620 23:29:28.669843 9029 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:28.670029 9029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:28.670450 9029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:28.674372 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "e7f8cd44c19c49979619131d7d460ac6"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "1a3fffeedebce480625b03ab76da3eec"
server_key_iv: "53df45fb1a8f36cd0f071d44a21d2e4b"
server_key_version: "encryptionkey@0"
I20260620 23:29:28.675436 9029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "e7f8cd44c19c49979619131d7d460ac6"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "1a3fffeedebce480625b03ab76da3eec"
server_key_iv: "53df45fb1a8f36cd0f071d44a21d2e4b"
server_key_version: "encryptionkey@0"
I20260620 23:29:28.682834 9029 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 23:29:28.687883 9044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:28.690168 9029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:28.690377 9029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "e7f8cd44c19c49979619131d7d460ac6"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "1a3fffeedebce480625b03ab76da3eec"
server_key_iv: "53df45fb1a8f36cd0f071d44a21d2e4b"
server_key_version: "encryptionkey@0"
I20260620 23:29:28.690636 9029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:28.714965 9029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:28.719538 9029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:28.719851 9029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:28.721088 9029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:28.722662 9029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:28.722744 9029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:28.722867 9029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:28.722934 9029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:28.761876 9029 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:33189
I20260620 23:29:28.762059 9156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:33189 every 8 connection(s)
I20260620 23:29:28.763309 9029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:28.771770 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9029
I20260620 23:29:28.771955 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:28.772590 387 external_mini_cluster.cc:1428] Setting key 3015d5c4f496ceaa487129815cf014c6
Jun 20 23:29:28 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:28.788331 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:28.766915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38743 (local address 127.0.96.254:38917)
0620 23:29:28.767475 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:28.767480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:28.768830 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 23:29:28.771131 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:28.771137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:28.771280 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:28.771452 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:28.773649 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.774784 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.777164 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.777642 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.777774 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:28.781500 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:28.781506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:28.781509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:28.781558 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:28.784397 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:28.785410 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:28.785416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:28.785421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:28.785619 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:28.786196 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:28.786203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:28.786205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:28.786546 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:28.786759 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:28.787597 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 23:29:28.787893 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":331,"threads_started":1}
I20260620 23:29:28.789415 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:28.767687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:38743)
0620 23:29:28.768609 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:28.768672 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:28.770791 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:28.771806 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:28.771819 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:28.772614 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:28.773441 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:28.773465 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.774992 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.775002 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:28.776908 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:28.776922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.777889 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.777894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:28.778105 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:28.778842 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:28.778865 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:28.781234 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:28.784567 (+ 3333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:28.784574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:28.784588 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:28.785252 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:28.785762 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:28.785766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:28.785768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:28.785965 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:28.786716 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:28.786724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:28.787086 (+ 362us) client_negotiation.cc:769] Sending connection context
0620 23:29:28.787398 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 23:29:28.787948 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":372,"threads_started":1}
I20260620 23:29:28.790874 9157 heartbeater.cc:347] Connected to a master server at 127.0.96.254:38917
I20260620 23:29:28.791195 9157 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:28.791925 9157 heartbeater.cc:511] Master 127.0.96.254:38917 requested a full tablet report, sending...
I20260620 23:29:28.793659 8700 ts_manager.cc:194] Registered new tserver with Master: e7f8cd44c19c49979619131d7d460ac6 (127.0.96.195:33189)
I20260620 23:29:28.794880 8700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:38743
I20260620 23:29:28.803216 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab.
Entry for principal oryx/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab.
WARNING: no policy specified for oryx/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab.
Entry for principal oryx/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab.
WARNING: no policy specified for oryx/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab.
Entry for principal oryx/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab.
I20260620 23:29:28.961578 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 8761
Jun 20 23:29:28 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:29:28.979729 9168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:28.962962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.0.96.193:39957)
0620 23:29:28.963714 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:28.963721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:28.963754 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:29:28.963828 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:28.963835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:28.964101 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:28.964281 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:28.965813 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.966907 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.968377 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:28.968942 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:28.969159 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:28.973667 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:28.973678 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:28.973694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:28.973734 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:28.976720 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:28.977573 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:28.977589 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:28.977591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:28.977738 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:28.978123 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:28.978128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:28.978130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:28.978475 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:28.978786 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:28.979072 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 23:29:28.979373 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":416,"threads_started":1}
I20260620 23:29:28.981292 8884 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:29.089959 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 8761
I20260620 23:29:29.113420 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 8895
Jun 20 23:29:29 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:29:29.130968 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:29.114494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.0.96.194:44635)
0620 23:29:29.115122 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:29.115129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:29.115157 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:29:29.115237 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:29.115243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:29.115468 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:29.115651 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:29.116886 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:29.117975 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:29.119271 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:29.119828 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:29.119966 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:29.124323 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:29.124334 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:29.124349 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:29.124385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:29.127730 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:29.128595 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:29.128602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:29.128605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:29.128741 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:29.129192 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:29.129199 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:29.129203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:29.129579 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:29.129942 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:29.130287 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 23:29:29.130579 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":334,"threads_started":1}
I20260620 23:29:29.132364 9018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:29.224129 9023 heartbeater.cc:503] Master 127.0.96.254:38917 was elected leader, sending a full tablet report...
I20260620 23:29:29.225595 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 8895
W20260620 23:29:29.243232 8684 connection.cc:535] server connection from 127.0.96.194:44915 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:29:29.243674 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9029
Jun 20 23:29:29 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:29:29.262677 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:29.244838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.0.96.195:33189)
0620 23:29:29.245720 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:29.245728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:29.245762 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:29:29.245829 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:29.245836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:29.246061 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:29.246261 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:29.247669 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:29.248646 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:29.250060 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:29.250615 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:29.250769 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:29.255807 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:29.255820 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:29.255837 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:29.255882 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:29.259254 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:29.260349 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:29.260355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:29.260358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:29.260495 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:29.260974 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:29.260977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:29.261031 (+ 54us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:29.261419 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:29.261657 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:29.262127 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 23:29:29.262372 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":522,"threads_started":1}
I20260620 23:29:29.264396 9152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:29.358568 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9029
I20260620 23:29:29.377043 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 8668
I20260620 23:29:29.378355 8730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:29.481451 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 8668
I20260620 23:29:29.507513 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38917
--webserver_interface=127.0.96.254
--webserver_port=38131
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:29.914523 9175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:29.915009 9175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:29.915197 9175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:29.923467 9175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:29.923700 9175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:29.923799 9175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:29.923915 9175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:29.924013 9175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:29.936024 9175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=38131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:29.938009 9175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:29.939747 9175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:29.954483 9180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:29.954530 9183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:29.954583 9181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:29.957422 9175 server_base.cc:1034] running on GCE node
I20260620 23:29:29.958437 9175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:29.960178 9175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:29.961474 9175 hybrid_clock.cc:648] HybridClock initialized: now 1781998169961410 us; error 61 us; skew 500 ppm
I20260620 23:29:29.961879 9175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:29 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:29.965384 9175 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:29:29.967200 9175 webserver.cc:460] Webserver started at http://127.0.96.254:38131/ using document root <none> and password file <none>
I20260620 23:29:29.968029 9175 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:29.968155 9175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:29.974468 9175 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:29:29.978153 9190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:29.980098 9175 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:29:29.980336 9175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5"
format_stamp: "Formatted at 2026-06-20 23:29:26 on dist-test-slave-v34r"
server_key: "e9864eac835f37a39752bd62cdef2824"
server_key_iv: "c3674d80958085e492a3d22f4fbd49aa"
server_key_version: "encryptionkey@0"
I20260620 23:29:29.981530 9175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:30.002300 9175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:30.007916 9175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:30.008268 9175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:30.035143 9175 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38917
I20260620 23:29:30.035182 9241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38917 every 8 connection(s)
I20260620 23:29:30.036927 9175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:30.046137 9242 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:29:30.046234 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9175
I20260620 23:29:30.047057 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:39957
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:38917
--webserver_port=41303
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:29:30.051149 9242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Bootstrap starting.
I20260620 23:29:30.076310 9242 log.cc:824] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:30.088081 9242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:29:30.088766 9242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Bootstrap complete.
I20260620 23:29:30.100389 9242 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:30.101642 9242 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2e8f7b3b0641fcb333c7d4c071e9d5, State: Initialized, Role: FOLLOWER
I20260620 23:29:30.102474 9242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:30.102778 9242 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:30.102981 9242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:30.103206 9242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:29:30.108016 9242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:30.108724 9242 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2e8f7b3b0641fcb333c7d4c071e9d5; no voters:
I20260620 23:29:30.109596 9242 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:29:30.109761 9246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:29:30.111198 9246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [term 2 LEADER]: Becoming Leader. State: Replica: 3c2e8f7b3b0641fcb333c7d4c071e9d5, State: Running, Role: LEADER
I20260620 23:29:30.111974 9246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } }
I20260620 23:29:30.112407 9242 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:30.114930 9248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2e8f7b3b0641fcb333c7d4c071e9d5. Latest consensus state: current_term: 2 leader_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } } }
I20260620 23:29:30.115418 9248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:30.115732 9247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2e8f7b3b0641fcb333c7d4c071e9d5" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38917 } } }
I20260620 23:29:30.116243 9247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:30.128139 9255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:30.135814 9255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:30.137466 9255 catalog_manager.cc:1259] Loaded cluster ID: 97099eb68692441a913b4d46a4849186
I20260620 23:29:30.137645 9255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:30.142556 9255 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:30.148500 9255 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c2e8f7b3b0641fcb333c7d4c071e9d5: Loaded TSK: 0
I20260620 23:29:30.150302 9255 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:29:30.422991 9244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:30.423319 9244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:30.423480 9244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:30.430935 9244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:30.431190 9244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:30.431461 9244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:30.441993 9244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.193.keytab
--principal=oryx/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:39957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=41303
--tserver_master_addrs=127.0.96.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:30.443462 9244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:30.444782 9244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:30.461340 9269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:30.461393 9268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:30.463858 9271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:30.464249 9244 server_base.cc:1034] running on GCE node
I20260620 23:29:30.465070 9244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:30.466203 9244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:30.467474 9244 hybrid_clock.cc:648] HybridClock initialized: now 1781998170467402 us; error 64 us; skew 500 ppm
I20260620 23:29:30.467810 9244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998170, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:30.471282 9244 init.cc:377] Logged in from keytab as oryx/127.0.96.193@KRBTEST.COM (short username oryx)
I20260620 23:29:30.473129 9244 webserver.cc:460] Webserver started at http://127.0.96.193:41303/ using document root <none> and password file <none>
I20260620 23:29:30.473970 9244 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:30.474110 9244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:30.480185 9244 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 23:29:30.483513 9278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:30.485509 9244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:29:30.485733 9244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "9898e145ac22454bba182bd5b47a4c1d"
format_stamp: "Formatted at 2026-06-20 23:29:27 on dist-test-slave-v34r"
server_key: "dd9916547f6cee1491d306813387e81a"
server_key_iv: "47cc3fa82d63f8e58652fc3e265a9e20"
server_key_version: "encryptionkey@0"
I20260620 23:29:30.486853 9244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:30.506232 9244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:30.511308 9244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:30.511600 9244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:30.512923 9244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:30.514590 9244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:30.514726 9244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:30.514866 9244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:30.514953 9244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:30.562384 9244 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:39957
I20260620 23:29:30.562446 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:39957 every 8 connection(s)
I20260620 23:29:30.563874 9244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:30.565785 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9244
I20260620 23:29:30.566538 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:44635
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:38917
--webserver_port=33737
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:30.588424 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:30.568149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:58747)
0620 23:29:30.570306 (+ 2157us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:30.570372 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:30.572481 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:30.574417 (+ 1936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:30.574431 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:30.575203 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:30.575962 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:30.575984 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.578122 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.578126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:30.579672 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:30.579687 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.580877 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.580886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:30.581259 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:30.582244 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:30.582291 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:30.587155 (+ 4864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1817,"thread_start_us":1586,"threads_started":1}
W20260620 23:29:30.589366 9391 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:38917 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.96.254:38917: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
W20260620 23:29:30.590466 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:30.568105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:58747 (local address 127.0.96.254:38917)
0620 23:29:30.569660 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:30.569675 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:30.570567 (+ 892us) server_negotiation.cc:407] Connection header received
0620 23:29:30.573293 (+ 2726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:30.573307 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:30.573752 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:30.574199 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:30.576259 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.577889 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.579866 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.580579 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.580837 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:30.587818 (+ 6981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:58747: BlockingRecv error: recv got EOF from 127.0.96.193:58747 (error 108)
Metrics: {"server-negotiator.queue_time_us":1276,"thread_start_us":1010,"threads_started":1}
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:30.601123 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:30.590182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:50799)
0620 23:29:30.590514 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:30.590538 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:30.590678 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:30.591587 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:30.591591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:30.592009 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:30.592207 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:30.592217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.593799 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.593805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:30.594827 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:30.594840 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.595809 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.595813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:30.595972 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:30.596743 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:30.596763 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:30.600779 (+ 4016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184}
W20260620 23:29:30.601359 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:30.590369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:50799 (local address 127.0.96.254:38917)
0620 23:29:30.591213 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:30.591222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:30.591250 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:29:30.591324 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:30.591330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:30.591417 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:30.591545 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:30.592368 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.593595 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.595000 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.595640 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.595776 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:30.601090 (+ 5314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:50799: BlockingRecv error: recv got EOF from 127.0.96.193:50799 (error 108)
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":901,"threads_started":1}
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:30.611423 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:30.602149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:48359)
0620 23:29:30.602356 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:30.602375 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:30.602498 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:30.602948 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:30.602951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:30.603275 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:30.603456 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:30.603464 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.604872 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.604876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:30.606007 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:30.606020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.606974 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.606978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:30.607172 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:30.607934 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:30.607955 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:30.611145 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:29:30.611727 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:30.602237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:48359 (local address 127.0.96.254:38917)
0620 23:29:30.602482 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:30.602489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:30.602512 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:29:30.602699 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:30.602704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:30.602794 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:30.602905 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:30.603611 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.604678 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.606170 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:30.606795 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:30.606938 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:30.611436 (+ 4498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:48359: BlockingRecv error: recv got EOF from 127.0.96.193:48359 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260620 23:29:30.611913 9391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:29:30.908077 9395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:30.908396 9395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:30.908536 9395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:30.916442 9395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:30.916599 9395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:30.916803 9395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:30.927011 9395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.194.keytab
--principal=oryx/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:44635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=33737
--tserver_master_addrs=127.0.96.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:30.928465 9395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:30.929793 9395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:30.944108 9401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:30.944406 9402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:30.944406 9404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:30.945576 9395 server_base.cc:1034] running on GCE node
I20260620 23:29:30.946376 9395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:30.947657 9395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:30.948932 9395 hybrid_clock.cc:648] HybridClock initialized: now 1781998170948854 us; error 76 us; skew 500 ppm
I20260620 23:29:30.949349 9395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:30 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998170, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:30.953217 9395 init.cc:377] Logged in from keytab as oryx/127.0.96.194@KRBTEST.COM (short username oryx)
I20260620 23:29:30.955430 9395 webserver.cc:460] Webserver started at http://127.0.96.194:33737/ using document root <none> and password file <none>
I20260620 23:29:30.956617 9395 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:30.956842 9395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:30.964304 9395 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:29:30.968662 9411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:30.970736 9395 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:30.971230 9395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "1c452d48dd6e4c86805de8407fb7f6c0"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "64223f74874f7cd3fcd0213b2cfb76d9"
server_key_iv: "bc9c4c3f83d7c40dcd39e92681ce4ee3"
server_key_version: "encryptionkey@0"
I20260620 23:29:30.972602 9395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:30.998037 9395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:31.004132 9395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:31.004531 9395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:31.005877 9395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:31.007690 9395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:31.007834 9395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:31.007993 9395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:31.008086 9395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:31.048552 9395 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:44635
I20260620 23:29:31.048632 9523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:44635 every 8 connection(s)
I20260620 23:29:31.050364 9395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:31.053525 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9395
I20260620 23:29:31.054256 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:33189
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:38917
--webserver_port=36507
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:29:31.071609 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.057276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:41299 (local address 127.0.96.254:38917)
0620 23:29:31.057553 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.057562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.057596 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:29:31.057918 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.057924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.058030 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.058173 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.060288 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.061192 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.064162 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.064635 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.064762 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.071364 (+ 6602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:41299: BlockingRecv error: recv got EOF from 127.0.96.194:41299 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:29:31.072404 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.054152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:41299)
0620 23:29:31.055228 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.055290 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.057512 (+ 2222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.058495 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.058505 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.059178 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.059947 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.059981 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.061400 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.061409 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.063929 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.063953 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.064956 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.064966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.065383 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.066338 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.066386 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.071194 (+ 4808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":457,"threads_started":1}
W20260620 23:29:31.073416 9524 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:38917 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.96.254:38917: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:31.083638 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.074256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:45205)
0620 23:29:31.074571 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.074588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.074691 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.075177 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.075181 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.075509 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.075669 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.075678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.077450 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.077454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.078598 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.078635 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.079849 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.079853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.079985 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.080645 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.080662 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.083433 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181}
W20260620 23:29:31.084115 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.074425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:45205 (local address 127.0.96.254:38917)
0620 23:29:31.074710 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.074720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.074752 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:29:31.074846 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.074854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.074969 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.075128 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.075859 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.077259 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.078800 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.079648 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.079835 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.083804 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:45205: BlockingRecv error: recv got EOF from 127.0.96.194:45205 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:31.092875 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.084636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:49313)
0620 23:29:31.084907 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.084933 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.085116 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.085581 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.085586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.085897 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.086136 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.086148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.087217 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.087223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.088111 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.088121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.088840 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.088845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.089052 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.089630 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.089647 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.092552 (+ 2905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260620 23:29:31.093214 9524 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:29:31.093346 9393 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.084837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:49313 (local address 127.0.96.254:38917)
0620 23:29:31.085070 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.085075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.085095 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:31.085312 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.085319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.085426 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.085545 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.086281 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.087082 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.088271 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.088693 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.088830 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.092755 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:49313: BlockingRecv error: recv got EOF from 127.0.96.194:49313 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 23:29:31.467078 9527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:31.467515 9527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:31.467670 9527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:31.475764 9527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:31.475986 9527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:31.476261 9527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:31.487946 9527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/krb5kdc/oryx_127.0.96.195.keytab
--principal=oryx/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:33189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=36507
--tserver_master_addrs=127.0.96.254:38917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:31.489823 9527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:31.491657 9527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:31.508375 9532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:31.508445 9535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:31.508425 9533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:31.514189 9527 server_base.cc:1034] running on GCE node
I20260620 23:29:31.515120 9527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:31.516495 9527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:31.517791 9527 hybrid_clock.cc:648] HybridClock initialized: now 1781998171517690 us; error 78 us; skew 500 ppm
I20260620 23:29:31.518273 9527 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998171, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:31.522140 9527 init.cc:377] Logged in from keytab as oryx/127.0.96.195@KRBTEST.COM (short username oryx)
I20260620 23:29:31.524117 9527 webserver.cc:460] Webserver started at http://127.0.96.195:36507/ using document root <none> and password file <none>
I20260620 23:29:31.525102 9527 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:31.525242 9527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:31.531980 9527 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:29:31.535641 9542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:31.537689 9527 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:31.537945 9527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "e7f8cd44c19c49979619131d7d460ac6"
format_stamp: "Formatted at 2026-06-20 23:29:28 on dist-test-slave-v34r"
server_key: "1a3fffeedebce480625b03ab76da3eec"
server_key_iv: "53df45fb1a8f36cd0f071d44a21d2e4b"
server_key_version: "encryptionkey@0"
I20260620 23:29:31.539207 9527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:31.559603 9527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:31.566834 9527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:31.567219 9527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:31.568699 9527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:31.570596 9527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:31.570700 9527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:31.570818 9527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:31.570996 9527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:31.612262 9527 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:33189
I20260620 23:29:31.612339 9654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:33189 every 8 connection(s)
I20260620 23:29:31.614629 9527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:31.615643 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9527
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 23:29:31.626844 9656 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.612927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:51171 (local address 127.0.96.254:38917)
0620 23:29:31.613609 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.613615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.613638 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:29:31.613706 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.613711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.613797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.613910 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.614795 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.615722 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.617959 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.618898 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.619045 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.626368 (+ 7323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:51171: BlockingRecv error: recv got EOF from 127.0.96.193:51171 (error 108)
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":421,"threads_started":1}
I20260620 23:29:31.628139 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.612764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:51171)
0620 23:29:31.613380 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.613405 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.613559 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.613940 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.613944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.614448 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.614641 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.614650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.615911 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.615917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.617756 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.617776 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.619335 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.619343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.619594 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.620471 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.620505 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.627646 (+ 7141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":301,"threads_started":1}
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:31.646801 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.617324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49694 (local address 127.0.96.254:38917)
0620 23:29:31.627078 (+ 9754us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.627085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.627110 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:29:31.627187 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.627195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.627293 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.627424 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.630465 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.631496 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.633275 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.633920 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.634074 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.638047 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:31.638058 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:31.638083 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:31.638145 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:31.642570 (+ 4425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:31.643539 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:31.643547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:31.643554 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:31.643780 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:31.644306 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:31.644313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:31.644317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:31.645068 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:31.645339 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:31.645939 (+ 600us) server_negotiation.cc:299] Negotiation successful
0620 23:29:31.646368 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9654,"thread_start_us":11466,"threads_started":1}
I20260620 23:29:31.648052 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.625961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:37661)
0620 23:29:31.631166 (+ 5205us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.631237 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.633260 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.634232 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.634245 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.634920 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.635609 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.635627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.637726 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.637731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.639875 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.639894 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.641028 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.641039 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.641417 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.642173 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.642206 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.647080 (+ 4874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4758,"thread_start_us":4422,"threads_started":1}
W20260620 23:29:31.648078 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.624731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:37661 (local address 127.0.96.254:38917)
0620 23:29:31.628774 (+ 4043us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.628784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.631410 (+ 2626us) server_negotiation.cc:407] Connection header received
0620 23:29:31.633660 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.633664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.633766 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.633882 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.635838 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.637521 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.640112 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.640668 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.640816 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.647710 (+ 6894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:37661: BlockingRecv error: recv got EOF from 127.0.96.195:37661 (error 108)
Metrics: {"server-negotiator.queue_time_us":3849,"thread_start_us":4427,"threads_started":1}
W20260620 23:29:31.649175 9657 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:38917 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.96.254:38917: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:31.659068 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.649919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:52449)
0620 23:29:31.650098 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.650146 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.650255 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.650675 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.650679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.651136 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.651380 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.651393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.652700 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.652705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.653795 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.653810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.654593 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.654597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.654764 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.655474 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.655498 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.658799 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 23:29:31.659396 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.650005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:52449 (local address 127.0.96.254:38917)
0620 23:29:31.650305 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.650311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.650331 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:31.650416 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.650421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.650497 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.650599 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.651604 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.652514 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.653974 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.654420 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.654574 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.659047 (+ 4473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:52449: BlockingRecv error: recv got EOF from 127.0.96.195:52449 (error 108)
Metrics: {"server-negotiator.queue_time_us":163}
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:31 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:31.669675 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:31.659903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:46745)
0620 23:29:31.660104 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:31.660122 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:31.660224 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:31.660613 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:31.660616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:31.660953 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:31.661246 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.661256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.662326 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.662368 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.663679 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:31.663702 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.664658 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.664663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:31.664888 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.665739 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:31.665760 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:31.669391 (+ 3631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 23:29:31.669890 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:31.659993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:46745 (local address 127.0.96.254:38917)
0620 23:29:31.660142 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:31.660145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:31.660166 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:29:31.660334 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:31.660337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:31.660409 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:31.660505 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:31.661391 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.662143 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.663877 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:31.664473 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:31.664617 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:31.669601 (+ 4984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:46745: BlockingRecv error: recv got EOF from 127.0.96.195:46745 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 23:29:31.670138 9657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 23:29:32 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:32 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:32.108646 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:32.094052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:54969)
0620 23:29:32.094734 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:32.094757 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:32.094936 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:32.095310 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:32.095315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:32.095815 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:32.096007 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:32.096016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.097064 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.097069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:32.098244 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:32.098257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.099164 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.099171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:32.099360 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:32.104821 (+ 5461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:32.104860 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:32.108351 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":371,"threads_started":1}
W20260620 23:29:32.108846 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:32.094151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:54969 (local address 127.0.96.254:38917)
0620 23:29:32.094485 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:32.094493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:32.094856 (+ 363us) server_negotiation.cc:407] Connection header received
0620 23:29:32.095069 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:32.095075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:32.095162 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:32.095277 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:32.096170 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.096875 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.098418 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.098905 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.099057 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:32.108588 (+ 9531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:54969: BlockingRecv error: recv got EOF from 127.0.96.194:54969 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Jun 20 23:29:32 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:32 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:32.638793 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:32.629449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:52433)
0620 23:29:32.630103 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:32.630124 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:32.630258 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:32.630713 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:32.630717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:32.631124 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:32.631277 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:32.631285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.632525 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.632528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:32.633496 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:32.633507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.634345 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.634352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:32.634554 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:32.635455 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:32.635478 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:32.638534 (+ 3056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":332,"threads_started":1}
W20260620 23:29:32.639242 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:32.629573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52433 (local address 127.0.96.254:38917)
0620 23:29:32.630194 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:32.630200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:32.630224 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:29:32.630396 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:32.630401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:32.630492 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:32.630613 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:32.631478 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.632354 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.633656 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.634121 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.634289 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:32.638902 (+ 4613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:52433: BlockingRecv error: recv got EOF from 127.0.96.193:52433 (error 108)
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":320,"threads_started":1}
Jun 20 23:29:32 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:32 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:32.682631 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:32.670933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:49909)
0620 23:29:32.671632 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:32.671657 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:32.671875 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:32.672273 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:32.672277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:32.672939 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:32.673302 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:32.673319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.675008 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.675014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:32.676554 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:32.676572 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.677583 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.677590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:32.677868 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:32.678691 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:32.678712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:32.682356 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":360,"threads_started":1}
W20260620 23:29:32.683034 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:32.671151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:49909 (local address 127.0.96.254:38917)
0620 23:29:32.671465 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:32.671481 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:32.671780 (+ 299us) server_negotiation.cc:407] Connection header received
0620 23:29:32.672040 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:32.672045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:32.672117 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:32.672232 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:32.673567 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.674800 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.676750 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:32.677341 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:32.677506 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:32.682718 (+ 5212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:49909: BlockingRecv error: recv got EOF from 127.0.96.195:49909 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 23:29:33 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:33 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:33.119987 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:33.109863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:38123)
0620 23:29:33.110454 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:33.110478 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:33.110608 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:33.110988 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:33.110991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:33.111395 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:33.111548 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:33.111555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.112798 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.112809 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:33.113938 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:33.113960 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.114781 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.114788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:33.114975 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:33.115680 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:33.115698 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:33.119576 (+ 3878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":338,"threads_started":1}
W20260620 23:29:33.120127 9665 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:33.109947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:38123 (local address 127.0.96.254:38917)
0620 23:29:33.110254 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:33.110262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:33.110575 (+ 313us) server_negotiation.cc:407] Connection header received
0620 23:29:33.110732 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:33.110737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:33.110818 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:33.110934 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:33.111686 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.112531 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.114113 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.114565 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.114734 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:33.119816 (+ 5082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:38123: BlockingRecv error: recv got EOF from 127.0.96.194:38123 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Jun 20 23:29:33 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:33 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:33.651736 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:33.640195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:55053)
0620 23:29:33.640831 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:33.640851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:33.641058 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:33.641745 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:33.641753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:33.642287 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:33.642555 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:33.642565 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.644171 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.644176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:33.645347 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:33.645363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.646403 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.646409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:33.646586 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:33.647690 (+ 1104us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:33.647713 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:33.651460 (+ 3747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":344,"threads_started":1}
W20260620 23:29:33.652252 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:33.640667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:55053 (local address 127.0.96.254:38917)
0620 23:29:33.641239 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:33.641245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:33.641264 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:29:33.641322 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:33.641328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:33.641440 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:33.641556 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:33.642764 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.643744 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.645553 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.646052 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.646184 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:33.651905 (+ 5721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:55053: BlockingRecv error: recv got EOF from 127.0.96.193:55053 (error 108)
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":337,"threads_started":1}
Jun 20 23:29:33 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:33 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:33.694164 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:33.683839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:34213)
0620 23:29:33.684449 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:33.684471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:33.684600 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:33.685379 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:33.685383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:33.685812 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:33.685971 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:33.685980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.687299 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.687304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:33.688301 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:33.688315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.689385 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.689390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:33.689519 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:33.690276 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:33.690293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:33.693742 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":375,"threads_started":1}
W20260620 23:29:33.694191 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:33.683887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:34213 (local address 127.0.96.254:38917)
0620 23:29:33.684185 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:33.684195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:33.684600 (+ 405us) server_negotiation.cc:407] Connection header received
0620 23:29:33.684836 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:33.684841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:33.684918 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:33.685334 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:33.686139 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.687115 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.688517 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:33.689202 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:33.689342 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:33.693983 (+ 4641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:34213: BlockingRecv error: recv got EOF from 127.0.96.195:34213 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 23:29:34 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:34 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:34.131749 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:34.121413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:60829)
0620 23:29:34.122057 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:34.122086 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:34.122264 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:34.122768 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:34.122774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:34.123238 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:34.123491 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:34.123504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.124701 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.124710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:34.125786 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:34.125800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.126646 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.126654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:34.126868 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:34.127733 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:34.127755 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:34.131316 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":366,"threads_started":1}
W20260620 23:29:34.131906 9669 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:34.121484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:60829 (local address 127.0.96.254:38917)
0620 23:29:34.121714 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:34.121723 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:34.122254 (+ 531us) server_negotiation.cc:407] Connection header received
0620 23:29:34.122457 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:34.122463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:34.122583 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:34.122727 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:34.123678 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.124504 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.125967 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.126455 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.126627 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:34.131608 (+ 4981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:60829: BlockingRecv error: recv got EOF from 127.0.96.194:60829 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 23:29:34 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:34 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:34.662211 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:34.653159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:56503)
0620 23:29:34.653729 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:34.653755 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:34.653904 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:34.654432 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:34.654437 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:34.654837 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:34.654981 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:34.654989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.656027 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.656035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:34.657040 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:34.657060 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.657758 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.657764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:34.657947 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:34.658616 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:34.658633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:34.661924 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":310,"threads_started":1}
W20260620 23:29:34.662523 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:34.653322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:56503 (local address 127.0.96.254:38917)
0620 23:29:34.654075 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:34.654083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:34.654103 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:34.654153 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:34.654156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:34.654261 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:34.654392 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:34.655124 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.655893 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.657182 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.657604 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.657736 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:34.662172 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:56503: BlockingRecv error: recv got EOF from 127.0.96.193:56503 (error 108)
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":462,"threads_started":1}
Jun 20 23:29:34 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:34 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:34.704860 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:34.695577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:38329)
0620 23:29:34.696346 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:34.696370 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:34.696561 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:34.696934 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:34.696938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:34.697339 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:34.697499 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:34.697508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.698526 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.698531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:34.699440 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:34.699451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.700142 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.700148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:34.700319 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:34.700941 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:34.700956 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:34.704430 (+ 3474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":475,"threads_started":1}
W20260620 23:29:34.704908 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:34.695721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38329 (local address 127.0.96.254:38917)
0620 23:29:34.696200 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:34.696206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:34.696486 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:29:34.696710 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:34.696714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:34.696774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:34.696906 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:34.697646 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.698383 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.699588 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:34.699987 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:34.700120 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:34.704612 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:38329: BlockingRecv error: recv got EOF from 127.0.96.195:38329 (error 108)
Metrics: {"server-negotiator.queue_time_us":231}
Jun 20 23:29:35 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:35 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:35.143114 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:35.133215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:52737)
0620 23:29:35.133898 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:35.133942 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:35.134208 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:35.134773 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:35.134779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:35.135249 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:35.135421 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:35.135431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.136577 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.136584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:35.137644 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:35.137665 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.138472 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.138478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:35.138687 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:35.139422 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:35.139442 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:35.142807 (+ 3365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"spinlock_wait_cycles":44288,"thread_start_us":420,"threads_started":1}
W20260620 23:29:35.143375 9673 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:35.133345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:52737 (local address 127.0.96.254:38917)
0620 23:29:35.133627 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:35.133635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:35.134074 (+ 439us) server_negotiation.cc:407] Connection header received
0620 23:29:35.134456 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:35.134464 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:35.134603 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:35.134726 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:35.135588 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.136423 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.137810 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.138304 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.138451 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:35.143078 (+ 4627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:52737: BlockingRecv error: recv got EOF from 127.0.96.194:52737 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 20 23:29:35 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:35 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:35.673353 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:35.663408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:38895)
0620 23:29:35.664311 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:35.664340 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:35.664481 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:35.664873 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:35.664876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:35.665339 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:35.665516 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:35.665525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.666594 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.666598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:35.667641 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:35.667654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.668507 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.668511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:35.668637 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:35.669383 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:35.669400 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:35.672975 (+ 3575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":482,"threads_started":1}
W20260620 23:29:35.673676 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:35.663513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:38895 (local address 127.0.96.254:38917)
0620 23:29:35.664422 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:35.664427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:35.664449 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:29:35.664614 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:35.664618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:35.664701 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:35.664814 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:35.665674 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.666448 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.667835 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.668341 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.668479 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:35.673322 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:38895: BlockingRecv error: recv got EOF from 127.0.96.193:38895 (error 108)
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":697,"threads_started":1}
Jun 20 23:29:35 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:35 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:35.716418 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:35.706301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:56687)
0620 23:29:35.706880 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:35.706905 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:35.707052 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:35.707436 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:35.707439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:35.707792 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:35.707954 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:35.707962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.709162 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.709166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:35.710354 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:35.710365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.711380 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.711387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:35.711608 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:35.712374 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:35.712394 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:35.716088 (+ 3694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":329,"threads_started":1}
W20260620 23:29:35.716881 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:35.706423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:56687 (local address 127.0.96.254:38917)
0620 23:29:35.706707 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:35.706715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:35.707022 (+ 307us) server_negotiation.cc:407] Connection header received
0620 23:29:35.707180 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:35.707185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:35.707263 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:35.707373 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:35.708116 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.708914 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.710492 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:35.711184 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:35.711344 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:35.716562 (+ 5218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:56687: BlockingRecv error: recv got EOF from 127.0.96.195:56687 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 23:29:36 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:36 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:36.154961 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:36.144486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:44735)
0620 23:29:36.145209 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:36.145235 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:36.145442 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:36.146014 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:36.146020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:36.146562 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:36.146750 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:36.146760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.147908 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.147915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:36.148912 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:36.148926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.150023 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.150029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:36.150187 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:36.150930 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:36.150949 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:36.154651 (+ 3702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":317,"threads_started":1}
W20260620 23:29:36.155197 9677 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:36.144817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:44735 (local address 127.0.96.254:38917)
0620 23:29:36.145129 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:36.145137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:36.145339 (+ 202us) server_negotiation.cc:407] Connection header received
0620 23:29:36.145737 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:36.145744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:36.145840 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:36.145975 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:36.146918 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.147738 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.149259 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.149824 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.149966 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:36.154898 (+ 4932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:44735: BlockingRecv error: recv got EOF from 127.0.96.194:44735 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Jun 20 23:29:36 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:36 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:36.684880 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:36.674288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:41965)
0620 23:29:36.674875 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:36.674896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:36.675032 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:36.675631 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:36.675636 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:36.676117 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:36.676330 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:36.676340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.677604 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.677609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:36.678913 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:36.678933 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.680008 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.680016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:36.680270 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:36.681238 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:36.681254 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:36.684639 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":329,"threads_started":1}
W20260620 23:29:36.685354 9681 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:36.674486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:41965 (local address 127.0.96.254:38917)
0620 23:29:36.675223 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:36.675227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:36.675242 (+ 15us) server_negotiation.cc:407] Connection header received
0620 23:29:36.675294 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:36.675298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:36.675371 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:36.675462 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:36.676526 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.677331 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.679109 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.679797 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.679948 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:36.685050 (+ 5102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:41965: BlockingRecv error: recv got EOF from 127.0.96.193:41965 (error 108)
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":544,"threads_started":1}
Jun 20 23:29:36 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:36 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:36.727452 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:36.717647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:57691)
0620 23:29:36.718268 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:36.718292 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:36.718440 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:36.718878 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:36.718883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:36.719303 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:36.719473 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:36.719481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.720705 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.720710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:36.722131 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:36.722149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.723091 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.723096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:36.723217 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:36.723893 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:36.723909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:36.726912 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":354,"threads_started":1}
W20260620 23:29:36.727589 9681 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:36.717673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:57691 (local address 127.0.96.254:38917)
0620 23:29:36.717989 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:36.718001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:36.718413 (+ 412us) server_negotiation.cc:407] Connection header received
0620 23:29:36.718575 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:36.718580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:36.718659 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:36.718768 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:36.719630 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.720497 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.722311 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:36.722938 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:36.723058 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:36.727180 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:57691: BlockingRecv error: recv got EOF from 127.0.96.195:57691 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 23:29:37 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:37 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:37.171649 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:37.156334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:38631)
0620 23:29:37.157095 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:37.157122 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:37.157282 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:37.157732 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:37.157736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:37.158239 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:37.158442 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:37.158453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.160126 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.160132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:37.161682 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:37.161703 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.162502 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.162508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:37.162647 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:37.167919 (+ 5272us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:37.167947 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:37.171295 (+ 3348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":437,"threads_started":1}
W20260620 23:29:37.171762 9681 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:37.156475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:38631 (local address 127.0.96.254:38917)
0620 23:29:37.156656 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:37.156660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:37.157287 (+ 627us) server_negotiation.cc:407] Connection header received
0620 23:29:37.157402 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:37.157406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:37.157498 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:37.157620 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:37.158708 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.159911 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.161854 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.162269 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.162425 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:37.171502 (+ 9077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:38631: BlockingRecv error: recv got EOF from 127.0.96.194:38631 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 23:29:37 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:37 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:37.696823 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:37.686008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:34339)
0620 23:29:37.686844 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:37.686876 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:37.687066 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:37.687441 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:37.687445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:37.687976 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:37.688178 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:37.688189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.689383 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.689390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:37.691076 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:37.691092 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.692160 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.692165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:37.692311 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:37.693092 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:37.693112 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:37.696402 (+ 3290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":292,"threads_started":1}
W20260620 23:29:37.697000 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:37.686168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:34339 (local address 127.0.96.254:38917)
0620 23:29:37.686974 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:37.686978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:37.686999 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:29:37.687208 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:37.687212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:37.687278 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:37.687416 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:37.688327 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.689155 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.691239 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.692001 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.692109 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:37.696619 (+ 4510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:34339: BlockingRecv error: recv got EOF from 127.0.96.193:34339 (error 108)
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":593,"threads_started":1}
Jun 20 23:29:37 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:37 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:37.738870 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:37.728920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:40989)
0620 23:29:37.729631 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:37.729656 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:37.729808 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:37.730232 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:37.730236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:37.730829 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:37.731085 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:37.731095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.732248 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.732253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:37.733339 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:37.733354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.734172 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.734177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:37.734326 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:37.735160 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:37.735182 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:37.738533 (+ 3351us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":372,"threads_started":1}
W20260620 23:29:37.738955 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:37.729124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:40989 (local address 127.0.96.254:38917)
0620 23:29:37.729344 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:37.729350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:37.729801 (+ 451us) server_negotiation.cc:407] Connection header received
0620 23:29:37.729930 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:37.729936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:37.730036 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:37.730164 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:37.731284 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.732061 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.733512 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:37.733960 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:37.734078 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:37.738738 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:40989: BlockingRecv error: recv got EOF from 127.0.96.195:40989 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 23:29:38 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:38 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:38.183285 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:38.172895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:48431)
0620 23:29:38.173586 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:38.173609 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:38.173736 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:38.174130 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:38.174133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:38.174553 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:38.174739 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:38.174748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.176020 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.176025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:38.177098 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:38.177119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.177953 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.177958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:38.178156 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:38.179054 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:38.179084 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:38.182876 (+ 3792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":292,"threads_started":1}
W20260620 23:29:38.183749 9685 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:38.173122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:48431 (local address 127.0.96.254:38917)
0620 23:29:38.173492 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:38.173497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:38.173696 (+ 199us) server_negotiation.cc:407] Connection header received
0620 23:29:38.173836 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:38.173840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:38.173930 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:38.174034 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:38.174931 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.175847 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.177276 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.177791 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.177899 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:38.183240 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:48431: BlockingRecv error: recv got EOF from 127.0.96.194:48431 (error 108)
Metrics: {"server-negotiator.queue_time_us":202}
Jun 20 23:29:38 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:38 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:38.707688 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:38.697968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:38407)
0620 23:29:38.698595 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:38.698617 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:38.698803 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:38.699242 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:38.699247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:38.699715 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:38.699905 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:38.699913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.701139 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.701145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:38.702432 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:38.702453 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.703409 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.703416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:38.703604 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:38.704295 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:38.704314 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:38.707439 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":303,"threads_started":1}
W20260620 23:29:38.708105 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:38.698194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:38407 (local address 127.0.96.254:38917)
0620 23:29:38.698856 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:38.698861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:38.698881 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:38.698937 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:38.698941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:38.699022 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:38.699128 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:38.700074 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.700922 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.702612 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.703200 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.703340 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:38.707789 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:38407: BlockingRecv error: recv got EOF from 127.0.96.193:38407 (error 108)
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":422,"threads_started":1}
Jun 20 23:29:38 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:38 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:38.749698 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:38.740261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:52379)
0620 23:29:38.741056 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:38.741082 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:38.741226 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:38.741548 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:38.741551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:38.742033 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:38.742234 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:38.742243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.743332 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.743336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:38.744512 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:38.744523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.745246 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.745251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:38.745384 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:38.746048 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:38.746066 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:38.749421 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":340,"threads_started":1}
W20260620 23:29:38.750109 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:38.740420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:52379 (local address 127.0.96.254:38917)
0620 23:29:38.740632 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:38.740636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:38.741171 (+ 535us) server_negotiation.cc:407] Connection header received
0620 23:29:38.741332 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:38.741336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:38.741405 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:38.741501 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:38.742363 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.743149 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.744648 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:38.745076 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:38.745178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:38.749790 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:52379: BlockingRecv error: recv got EOF from 127.0.96.195:52379 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 20 23:29:39 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:39 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:39.194777 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:39.184375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:50099)
0620 23:29:39.185299 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:39.185331 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:39.185536 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:39.186012 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:39.186017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:39.186644 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:39.186881 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:39.186894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.188399 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.188404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:39.189504 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:39.189518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.190466 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.190472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:39.190622 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:39.191381 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:39.191400 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:39.194466 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":462,"threads_started":1}
W20260620 23:29:39.194974 9689 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:39.184445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:50099 (local address 127.0.96.254:38917)
0620 23:29:39.184804 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:39.184813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:39.185463 (+ 650us) server_negotiation.cc:407] Connection header received
0620 23:29:39.185723 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:39.185731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:39.185829 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:39.185980 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:39.187087 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.188233 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.189663 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.190226 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.190406 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:39.194690 (+ 4284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:50099: BlockingRecv error: recv got EOF from 127.0.96.194:50099 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 23:29:39 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:39 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:39.719276 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:39.708854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:42995)
0620 23:29:39.709611 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:39.709637 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:39.709809 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:39.710198 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:39.710203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:39.710694 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:39.710875 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:39.710884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.711955 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.711960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:39.713119 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:39.713132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.714140 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.714145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:39.714289 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:39.714989 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:39.715007 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:39.719013 (+ 4006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":381,"threads_started":1}
W20260620 23:29:39.719624 9693 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:39.708918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:42995 (local address 127.0.96.254:38917)
0620 23:29:39.709788 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:39.709794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:39.709816 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:29:39.709959 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:39.709965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:39.710038 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:39.710161 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:39.711020 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.711804 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.713382 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.713937 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.714076 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:39.719307 (+ 5231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:42995: BlockingRecv error: recv got EOF from 127.0.96.193:42995 (error 108)
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":633,"threads_started":1}
Jun 20 23:29:39 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:39 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:39.761116 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:39.750939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:51393)
0620 23:29:39.751556 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:39.751580 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:39.751748 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:39.752172 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:39.752176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:39.752640 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:39.752836 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:39.752846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.754220 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.754225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:39.755613 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:39.755631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.756554 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.756560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:39.756695 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:39.757451 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:39.757470 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:39.760749 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":317,"threads_started":1}
W20260620 23:29:39.761314 9693 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:39.751091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:51393 (local address 127.0.96.254:38917)
0620 23:29:39.751347 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:39.751352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:39.751696 (+ 344us) server_negotiation.cc:407] Connection header received
0620 23:29:39.751881 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:39.751885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:39.751962 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:39.752086 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:39.753027 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.753995 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.755782 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:39.756334 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:39.756456 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:39.760970 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:51393: BlockingRecv error: recv got EOF from 127.0.96.195:51393 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 23:29:40 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:40 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:40.206065 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:40.195942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:46867)
0620 23:29:40.196629 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:40.196654 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:40.196801 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:40.197268 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:40.197271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:40.197787 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:40.197999 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:40.198009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.199042 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.199046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:40.200065 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:40.200077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.200798 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.200802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:40.200939 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:40.201729 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:40.201747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:40.205660 (+ 3913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":393,"threads_started":1}
W20260620 23:29:40.206308 9693 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:40.196087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:46867 (local address 127.0.96.254:38917)
0620 23:29:40.196274 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:40.196279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:40.196776 (+ 497us) server_negotiation.cc:407] Connection header received
0620 23:29:40.196935 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:40.196939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:40.197109 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:40.197222 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:40.198152 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.198910 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.200204 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.200653 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.200755 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:40.205961 (+ 5206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:46867: BlockingRecv error: recv got EOF from 127.0.96.194:46867 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 23:29:40 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:40 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:40.730446 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:40.720517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:57759)
0620 23:29:40.721170 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:40.721189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:40.721320 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:40.721796 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:40.721800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:40.722296 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:40.722521 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:40.722535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.723844 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.723855 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:40.724835 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:40.724856 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.725655 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.725661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:40.725838 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:40.726825 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:40.726845 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:40.730165 (+ 3320us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":326,"threads_started":1}
W20260620 23:29:40.730731 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:40.720665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:57759 (local address 127.0.96.254:38917)
0620 23:29:40.721309 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:40.721314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:40.721338 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:29:40.721464 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:40.721468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:40.721604 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:40.721770 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:40.722716 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.723671 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.725039 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.725474 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.725607 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:40.730458 (+ 4851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:57759: BlockingRecv error: recv got EOF from 127.0.96.193:57759 (error 108)
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":364,"threads_started":1}
Jun 20 23:29:40 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:40 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:40.773062 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:40.762337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:37627)
0620 23:29:40.762997 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:40.763024 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:40.763210 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:40.763688 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:40.763692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:40.764196 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:40.764418 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:40.764429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.765667 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.765673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:40.766764 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:40.766781 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.767611 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.767617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:40.767785 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:40.768682 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:40.768704 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:40.772637 (+ 3933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":374,"threads_started":1}
W20260620 23:29:40.773161 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:40.762453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:37627 (local address 127.0.96.254:38917)
0620 23:29:40.762764 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:40.762770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:40.763176 (+ 406us) server_negotiation.cc:407] Connection header received
0620 23:29:40.763331 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:40.763335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:40.763426 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:40.763533 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:40.764601 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.765411 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.766946 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:40.767371 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:40.767483 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:40.772916 (+ 5433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:37627: BlockingRecv error: recv got EOF from 127.0.96.195:37627 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Jun 20 23:29:41 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:41 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:41.216478 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:41.207471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:42921)
0620 23:29:41.208111 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:41.208134 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:41.208309 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:41.208673 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:41.208677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:41.209208 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:41.209354 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:41.209362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.210613 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.210619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:41.211599 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:41.211611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.212305 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.212310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:41.212612 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:41.213356 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:41.213374 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:41.216089 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":386,"threads_started":1}
W20260620 23:29:41.216576 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:41.207576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:42921 (local address 127.0.96.254:38917)
0620 23:29:41.207872 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:41.207882 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:41.208238 (+ 356us) server_negotiation.cc:407] Connection header received
0620 23:29:41.208447 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:41.208456 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:41.208529 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:41.208637 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:41.209535 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.210467 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.211747 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.212147 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.212276 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:41.216311 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:42921: BlockingRecv error: recv got EOF from 127.0.96.194:42921 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 23:29:41 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:41 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:41.740911 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:41.731621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:38905)
0620 23:29:41.732275 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:41.732306 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:41.732470 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:41.732886 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:41.732890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:41.733404 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:41.733607 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:41.733616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.734685 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.734692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:41.735664 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:41.735675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.736454 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.736460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:41.736644 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:41.737411 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:41.737429 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:41.740647 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":347,"threads_started":1}
W20260620 23:29:41.741421 9701 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:41.731828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:38905 (local address 127.0.96.254:38917)
0620 23:29:41.732560 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:41.732566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:41.732586 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:41.732653 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:41.732657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:41.732733 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:41.732855 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:41.733758 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.734547 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.735819 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.736277 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.736433 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:41.741081 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:38905: BlockingRecv error: recv got EOF from 127.0.96.193:38905 (error 108)
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":484,"threads_started":1}
Jun 20 23:29:41 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:41 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:41.783999 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:41.774388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:33475)
0620 23:29:41.775145 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:41.775177 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:41.775345 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:41.775741 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:41.775745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:41.776161 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:41.776346 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:41.776354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.777647 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.777652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:41.778694 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:41.778706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.779395 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.779399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:41.779530 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:41.780275 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:41.780300 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:41.783716 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":498,"threads_started":1}
W20260620 23:29:41.784162 9701 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:41.774681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:33475 (local address 127.0.96.254:38917)
0620 23:29:41.774942 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:41.774948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:41.775315 (+ 367us) server_negotiation.cc:407] Connection header received
0620 23:29:41.775488 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:41.775492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:41.775563 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:41.775661 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:41.776519 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.777484 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.778849 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:41.779249 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:41.779347 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:41.783902 (+ 4555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:33475: BlockingRecv error: recv got EOF from 127.0.96.195:33475 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 23:29:42 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:42 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:42.232613 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:42.217807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:60355)
0620 23:29:42.218401 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:42.218423 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:42.218588 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:42.218903 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:42.218907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:42.219331 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:42.219527 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:42.219536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.220534 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.220542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:42.221490 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:42.221501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.222180 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.222186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:42.222359 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:42.228548 (+ 6189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:42.228569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:42.232270 (+ 3701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":329,"threads_started":1}
W20260620 23:29:42.232816 9701 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:42.217910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:60355 (local address 127.0.96.254:38917)
0620 23:29:42.218200 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:42.218208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:42.218523 (+ 315us) server_negotiation.cc:407] Connection header received
0620 23:29:42.218711 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:42.218716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:42.218778 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:42.218875 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:42.219665 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.220399 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.221629 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.222027 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.222157 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:42.232550 (+ 10393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:60355: BlockingRecv error: recv got EOF from 127.0.96.194:60355 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Jun 20 23:29:42 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:42 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:42.754006 9704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:42.742487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:54671)
0620 23:29:42.743700 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:42.743727 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:42.744020 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:42.744526 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:42.744531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:42.745148 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:42.745333 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:42.745342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.746549 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.746555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:42.747738 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:42.747753 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.748611 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.748617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:42.748813 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:42.749746 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:42.749767 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:42.753489 (+ 3722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1056,"thread_start_us":859,"threads_started":1}
W20260620 23:29:42.754377 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:42.742628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:54671 (local address 127.0.96.254:38917)
0620 23:29:42.743312 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:42.743322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:42.743939 (+ 617us) server_negotiation.cc:407] Connection header received
0620 23:29:42.744200 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:42.744207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:42.744359 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:42.744483 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:42.745485 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.746388 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.747925 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.748429 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.748579 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:42.753932 (+ 5353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:54671: BlockingRecv error: recv got EOF from 127.0.96.193:54671 (error 108)
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":319,"threads_started":1}
Jun 20 23:29:42 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:42 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:42.794490 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:42.785226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:57591)
0620 23:29:42.785783 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:42.785809 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:42.785977 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:42.786392 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:42.786395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:42.786873 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:42.787067 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:42.787076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.788257 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.788261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:42.789402 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:42.789415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.790263 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.790267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:42.790387 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:42.791181 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:42.791205 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:42.794235 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":306,"threads_started":1}
W20260620 23:29:42.794813 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:42.785351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:57591 (local address 127.0.96.254:38917)
0620 23:29:42.785554 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:42.785559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:42.785965 (+ 406us) server_negotiation.cc:407] Connection header received
0620 23:29:42.786120 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:42.786125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:42.786225 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:42.786350 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:42.787238 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.788052 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.789573 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:42.790082 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:42.790205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:42.794497 (+ 4292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:57591: BlockingRecv error: recv got EOF from 127.0.96.195:57591 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 23:29:43 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:43 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:43.243142 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:43.233713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:59673)
0620 23:29:43.234363 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:43.234397 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:43.234586 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:43.234945 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:43.234951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:43.235355 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:43.235510 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:43.235518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.236563 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.236569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:43.237563 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:43.237574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.238293 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.238299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:43.238474 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:43.239194 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:43.239210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:43.242724 (+ 3514us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":365,"threads_started":1}
W20260620 23:29:43.243275 9705 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:43.233820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:59673 (local address 127.0.96.254:38917)
0620 23:29:43.234116 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:43.234126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:43.234513 (+ 387us) server_negotiation.cc:407] Connection header received
0620 23:29:43.234731 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:43.234736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:43.234797 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:43.234907 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:43.235663 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.236426 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.237699 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.238141 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.238272 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:43.242932 (+ 4660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:59673: BlockingRecv error: recv got EOF from 127.0.96.194:59673 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 23:29:43 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:43 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:43.765261 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:43.755537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:44097)
0620 23:29:43.756245 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:43.756265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:43.756430 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:43.756904 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:43.756908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:43.757374 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:43.757552 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:43.757561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.758918 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.758922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:43.760065 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:43.760077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.760869 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.760873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:43.761073 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:43.761741 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:43.761761 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:43.764925 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":373,"threads_started":1}
W20260620 23:29:43.765597 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:43.755645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:44097 (local address 127.0.96.254:38917)
0620 23:29:43.756283 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:43.756288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:43.756372 (+ 84us) server_negotiation.cc:407] Connection header received
0620 23:29:43.756592 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:43.756605 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:43.756728 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:43.756838 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:43.757725 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.758718 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.760216 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.760698 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.760810 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:43.765263 (+ 4453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:44097: BlockingRecv error: recv got EOF from 127.0.96.193:44097 (error 108)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":363,"threads_started":1}
Jun 20 23:29:43 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:43 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:43.805537 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:43.795663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:46011)
0620 23:29:43.796542 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:43.796566 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:43.796754 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:43.797204 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:43.797209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:43.797712 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:43.797892 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:43.797901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.798981 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.798989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:43.799959 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:43.799970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.800740 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.800746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:43.800953 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:43.801732 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:43.801751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:43.805146 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":566,"threads_started":1}
W20260620 23:29:43.805683 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:43.795794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:46011 (local address 127.0.96.254:38917)
0620 23:29:43.796045 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:43.796053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:43.796673 (+ 620us) server_negotiation.cc:407] Connection header received
0620 23:29:43.796896 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:43.796907 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:43.797038 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:43.797160 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:43.798046 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.798826 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.800111 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:43.800577 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:43.800719 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:43.805416 (+ 4697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:46011: BlockingRecv error: recv got EOF from 127.0.96.195:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 23:29:44 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:44 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:44.254655 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:44.244331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:33711)
0620 23:29:44.245063 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:44.245086 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:44.245215 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:44.245624 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:44.245627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:44.246036 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:44.246215 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:44.246223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.247571 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.247577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:44.248885 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:44.248903 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.249839 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.249847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:44.250125 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:44.250917 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:44.250937 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:44.254388 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":429,"threads_started":1}
W20260620 23:29:44.255120 9709 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:44.244735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:33711 (local address 127.0.96.254:38917)
0620 23:29:44.244943 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:44.244953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:44.245223 (+ 270us) server_negotiation.cc:407] Connection header received
0620 23:29:44.245322 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:44.245326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:44.245435 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:44.245594 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:44.246423 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.247323 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.249128 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.249610 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.249764 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:44.254794 (+ 5030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:33711: BlockingRecv error: recv got EOF from 127.0.96.194:33711 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 23:29:44 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:44 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:44.776032 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:44.766394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:35593)
0620 23:29:44.766985 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:44.767004 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:44.767123 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:44.767537 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:44.767541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:44.767896 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:44.768090 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:44.768098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.769464 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.769468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:44.770394 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:44.770405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.771528 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.771533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:44.771663 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:44.772371 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:44.772388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:44.775640 (+ 3252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":317,"threads_started":1}
W20260620 23:29:44.776297 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:44.766527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:35593 (local address 127.0.96.254:38917)
0620 23:29:44.767114 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:44.767119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:44.767139 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:29:44.767229 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:44.767233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:44.767317 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:44.767429 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:44.768322 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.769250 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.770588 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.771328 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.771483 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:44.775987 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:35593: BlockingRecv error: recv got EOF from 127.0.96.193:35593 (error 108)
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":328,"threads_started":1}
Jun 20 23:29:44 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:44 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:44.816795 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:44.806893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:52469)
0620 23:29:44.807558 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:44.807582 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:44.807776 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:44.808174 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:44.808179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:44.808656 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:44.808836 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:44.808846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.810020 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.810026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:44.811022 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:44.811034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.811840 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.811847 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:44.812023 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:44.812764 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:44.812783 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:44.816477 (+ 3694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":334,"threads_started":1}
W20260620 23:29:44.817209 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:44.807045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:52469 (local address 127.0.96.254:38917)
0620 23:29:44.807397 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:44.807406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:44.807688 (+ 282us) server_negotiation.cc:407] Connection header received
0620 23:29:44.807928 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:44.807934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:44.808013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:44.808138 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:44.809028 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.809865 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.811177 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:44.811637 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:44.811790 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:44.816733 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:52469: BlockingRecv error: recv got EOF from 127.0.96.195:52469 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Jun 20 23:29:45 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:45 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:45.265178 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:45.255857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:47007)
0620 23:29:45.256449 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:45.256476 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:45.256622 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:45.257061 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:45.257064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:45.257419 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:45.257599 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:45.257607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.258844 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.258851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:45.259981 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:45.259996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.260821 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.260827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:45.261040 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:45.261825 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:45.261844 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:45.264835 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":314,"threads_started":1}
W20260620 23:29:45.265494 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:45.255995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:47007 (local address 127.0.96.254:38917)
0620 23:29:45.256286 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:45.256295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:45.256585 (+ 290us) server_negotiation.cc:407] Connection header received
0620 23:29:45.256749 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:45.256754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:45.256871 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:45.257041 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:45.257746 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.258639 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.260165 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.260631 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.260807 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:45.265151 (+ 4344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:47007: BlockingRecv error: recv got EOF from 127.0.96.194:47007 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 20 23:29:45 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:45 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.193@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:45.786274 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:45.777452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.193:58569)
0620 23:29:45.778213 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:45.778238 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:45.778368 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:45.778702 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:45.778707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:45.779192 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:45.779372 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:45.779379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.780557 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.780563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:45.781483 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:45.781493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.782171 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.782176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:45.782343 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:45.783016 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:45.783032 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:45.785900 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":384,"threads_started":1}
W20260620 23:29:45.786427 9717 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:45.777608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:58569 (local address 127.0.96.254:38917)
0620 23:29:45.778247 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:45.778254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:45.778295 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:29:45.778490 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:45.778495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:45.778573 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:45.778681 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:45.779512 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.780416 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.781640 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.782024 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.782160 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:45.786146 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.193:58569: BlockingRecv error: recv got EOF from 127.0.96.193:58569 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":374,"threads_started":1}
Jun 20 23:29:45 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:45 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.195@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:45.827142 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:45.817871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.195:56443)
0620 23:29:45.818419 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:45.818446 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:45.818609 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:45.819010 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:45.819014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:45.819438 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:45.819654 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:45.819663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.820750 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.820755 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:45.821942 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:45.821957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.822682 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.822686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:45.822827 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:45.823500 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:45.823517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:45.826788 (+ 3271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
W20260620 23:29:45.827395 9717 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:45.818058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:56443 (local address 127.0.96.254:38917)
0620 23:29:45.818298 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:45.818304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:45.818555 (+ 251us) server_negotiation.cc:407] Connection header received
0620 23:29:45.818720 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:45.818724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:45.818793 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:45.818892 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:45.819807 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.820565 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.822092 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:45.822504 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:45.822644 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:45.827105 (+ 4461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.195:56443: BlockingRecv error: recv got EOF from 127.0.96.195:56443 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 23:29:46 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 23:29:46 dist-test-slave-v34r krb5kdc[8655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.96.194@KRBTEST.COM for oryx/127.0.96.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 23:29:46.274781 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:46.266296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:38917 (local address 127.0.96.194:38739)
0620 23:29:46.266812 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:46.266832 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:46.266953 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:46.267329 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:46.267332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:46.267738 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:46.267951 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:46.267959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.269128 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.269133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:46.270151 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:46.270164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.270985 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.270990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:46.271126 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:46.271797 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:46.271814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:46.274519 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.96.254:38917: Server oryx/127.0.96.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":295,"threads_started":1}
W20260620 23:29:46.275344 9717 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:46.266427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:38739 (local address 127.0.96.254:38917)
0620 23:29:46.266656 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:46.266661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:46.266916 (+ 255us) server_negotiation.cc:407] Connection header received
0620 23:29:46.267086 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:46.267091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:46.267166 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:46.267290 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:46.268091 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.268887 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.270319 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.270802 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.270948 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:46.274955 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.96.194:38739: BlockingRecv error: recv got EOF from 127.0.96.194:38739 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:29:46.623503 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9244
I20260620 23:29:46.637463 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:46.625051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.96.193:39957)
0620 23:29:46.625740 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:46.625748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:46.625779 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:29:46.625851 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:46.625858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:46.626081 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:46.626258 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:46.627940 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.629067 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.630382 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.630979 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.631096 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:46.634279 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:46.634288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:46.634303 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:46.634342 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:46.636879 (+ 2537us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.96.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:29:46.637222 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37900: Request ticket server kudu/127.0.96.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":348,"threads_started":1}
W20260620 23:29:46.637584 9721 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37900: Request ticket server kudu/127.0.96.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:29:46.637893 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9244: Not authorized: Client connection negotiation failed: client connection to 127.0.96.193:39957: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.96.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:29:46.638144 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9244
I20260620 23:29:46.658942 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9395
I20260620 23:29:46.673307 9722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:46.660312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43032 (local address 127.0.96.194:44635)
0620 23:29:46.661102 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:46.661111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:46.661154 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:29:46.661238 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:46.661247 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:46.661549 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:46.661779 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:46.663088 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.664272 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.665652 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.666322 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.666455 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:46.669314 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:46.669326 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:46.669345 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:46.669397 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:46.672299 (+ 2902us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.96.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:29:46.672671 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43032: Request ticket server kudu/127.0.96.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":345,"threads_started":1}
W20260620 23:29:46.673323 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9395: Not authorized: Client connection negotiation failed: client connection to 127.0.96.194:44635: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.96.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:29:46.673436 9722 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43032: Request ticket server kudu/127.0.96.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:29:46.673525 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9395
I20260620 23:29:46.693424 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9527
I20260620 23:29:46.706707 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:46.694586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.0.96.195:33189)
0620 23:29:46.695216 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:46.695223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:46.695252 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:29:46.695319 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:46.695325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:46.695544 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:46.695767 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:46.697414 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.698507 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.699899 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:46.700443 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:46.700572 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:46.703478 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:46.703487 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:46.703501 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:46.703571 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:46.706184 (+ 2613us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.96.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 23:29:46.706460 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38214: Request ticket server kudu/127.0.96.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":346,"threads_started":1}
W20260620 23:29:46.706837 9723 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38214: Request ticket server kudu/127.0.96.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 23:29:46.707165 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9527: Not authorized: Client connection negotiation failed: client connection to 127.0.96.195:33189: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.96.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 23:29:46.707367 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9527
I20260620 23:29:46.724362 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9175
I20260620 23:29:46.725975 9237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:46.861997 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9175
2026-06-20T23:29:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22039 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T23:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:46Z Disabled control of system clock
I20260620 23:29:46.955598 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:39949
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:39949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 23:29:47.372977 9728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:47.373481 9728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:47.373620 9728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:47.383291 9728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:47.383445 9728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:47.383512 9728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:47.383594 9728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:47.383683 9728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:47.393232 9728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:39949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:47.394701 9728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:47.396642 9728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:47.409948 9736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:47.410013 9734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:47.410171 9733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:47.411372 9728 server_base.cc:1034] running on GCE node
I20260620 23:29:47.412317 9728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:47.414077 9728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:47.415388 9728 hybrid_clock.cc:648] HybridClock initialized: now 1781998187415306 us; error 76 us; skew 500 ppm
I20260620 23:29:47.415881 9728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:47.423256 9728 webserver.cc:460] Webserver started at http://127.0.96.254:34701/ using document root <none> and password file <none>
I20260620 23:29:47.424104 9728 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:47.424242 9728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:47.424613 9728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:47.427740 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "0971ab5ea40343c38ef4643033d2aebe"
format_stamp: "Formatted at 2026-06-20 23:29:47 on dist-test-slave-v34r"
server_key: "6ce6fe6ce603a5fb5a3a603b862522c3"
server_key_iv: "2ef1259a16be7b7a5a6b84cb2be9523a"
server_key_version: "encryptionkey@0"
I20260620 23:29:47.428560 9728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "0971ab5ea40343c38ef4643033d2aebe"
format_stamp: "Formatted at 2026-06-20 23:29:47 on dist-test-slave-v34r"
server_key: "6ce6fe6ce603a5fb5a3a603b862522c3"
server_key_iv: "2ef1259a16be7b7a5a6b84cb2be9523a"
server_key_version: "encryptionkey@0"
I20260620 23:29:47.434448 9728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 23:29:47.438766 9742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:47.440430 9728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:29:47.440654 9728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "0971ab5ea40343c38ef4643033d2aebe"
format_stamp: "Formatted at 2026-06-20 23:29:47 on dist-test-slave-v34r"
server_key: "6ce6fe6ce603a5fb5a3a603b862522c3"
server_key_iv: "2ef1259a16be7b7a5a6b84cb2be9523a"
server_key_version: "encryptionkey@0"
I20260620 23:29:47.440929 9728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:47.456877 9728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:47.457806 9728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:47.458094 9728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:47.478801 9728 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:39949
I20260620 23:29:47.478827 9793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:39949 every 8 connection(s)
I20260620 23:29:47.480228 9728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:47.485061 9794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:47.486330 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9728
I20260620 23:29:47.486536 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:47.487215 387 external_mini_cluster.cc:1428] Setting key 46ccd446cc298fd170104a11ac0f08e9
I20260620 23:29:47.503669 9794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe: Bootstrap starting.
I20260620 23:29:47.507236 9797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:47.489681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.0.96.254:39949)
0620 23:29:47.491115 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:47.491140 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:47.491233 (+ 93us) server_negotiation.cc:407] Connection header received
0620 23:29:47.493269 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:47.493314 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:47.494266 (+ 952us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:47.494909 (+ 643us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:47.495578 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:47.497770 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:47.499447 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:47.500870 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:47.501179 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:47.502982 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:47.502996 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:47.503029 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:47.504018 (+ 989us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:47.504172 (+ 154us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:47.504208 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:47.504372 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:47.504671 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 23:29:47.505180 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":394,"threads_started":1}
I20260620 23:29:47.509114 9794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:47.510536 9794 log.cc:824] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:47.514616 9794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe: No bootstrap required, opened a new log
I20260620 23:29:47.524048 9794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0971ab5ea40343c38ef4643033d2aebe" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39949 } }
I20260620 23:29:47.524597 9794 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:47.524705 9794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0971ab5ea40343c38ef4643033d2aebe, State: Initialized, Role: FOLLOWER
I20260620 23:29:47.525498 9794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0971ab5ea40343c38ef4643033d2aebe" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39949 } }
I20260620 23:29:47.525771 9794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:47.525920 9794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:47.526120 9794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:47.528824 9794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0971ab5ea40343c38ef4643033d2aebe" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39949 } }
I20260620 23:29:47.529707 9794 leader_election.cc:302] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0971ab5ea40343c38ef4643033d2aebe; no voters:
I20260620 23:29:47.530529 9794 leader_election.cc:288] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:47.530797 9799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:47.531752 9799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [term 1 LEADER]: Becoming Leader. State: Replica: 0971ab5ea40343c38ef4643033d2aebe, State: Running, Role: LEADER
I20260620 23:29:47.532408 9799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0971ab5ea40343c38ef4643033d2aebe" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39949 } }
I20260620 23:29:47.533567 9794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:47.535461 9800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0971ab5ea40343c38ef4643033d2aebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0971ab5ea40343c38ef4643033d2aebe" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39949 } } }
I20260620 23:29:47.535902 9800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:47.536914 9801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0971ab5ea40343c38ef4643033d2aebe. Latest consensus state: current_term: 1 leader_uuid: "0971ab5ea40343c38ef4643033d2aebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0971ab5ea40343c38ef4643033d2aebe" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 39949 } } }
I20260620 23:29:47.537461 9801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:47.539144 9804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:47.545481 9804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:47.556123 9804 catalog_manager.cc:1347] Generated new cluster ID: c2c2363f3c334732887e82b1eaefafa4
I20260620 23:29:47.556272 9804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:47.575832 9804 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:47.577282 9804 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:47.586452 9804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0971ab5ea40343c38ef4643033d2aebe: Generated new TSK 0
I20260620 23:29:47.587860 9804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:29:47.595443 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39949
--builtin_ntp_servers=127.0.96.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 23:29:47.957459 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:47.957756 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:47.957840 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:47.965212 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:47.965328 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:47.965445 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:47.974618 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:47.975880 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:47.977604 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:47.991549 9823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:47.991698 9826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:47.991716 9824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:47.992424 9818 server_base.cc:1034] running on GCE node
I20260620 23:29:47.993309 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:47.994534 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:47.995828 9818 hybrid_clock.cc:648] HybridClock initialized: now 1781998187995768 us; error 43 us; skew 500 ppm
I20260620 23:29:47.996279 9818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:47.999843 9818 webserver.cc:460] Webserver started at http://127.0.96.193:40329/ using document root <none> and password file <none>
I20260620 23:29:48.000808 9818 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:48.000964 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:48.001412 9818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:48.005276 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "878b3ef70c624729a82e421c36ce5321"
format_stamp: "Formatted at 2026-06-20 23:29:47 on dist-test-slave-v34r"
server_key: "dd331186f8648f3a17f65bbbb36e069c"
server_key_iv: "92bf2d4288c3b86bf008f490a699c806"
server_key_version: "encryptionkey@0"
I20260620 23:29:48.006340 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "878b3ef70c624729a82e421c36ce5321"
format_stamp: "Formatted at 2026-06-20 23:29:47 on dist-test-slave-v34r"
server_key: "dd331186f8648f3a17f65bbbb36e069c"
server_key_iv: "92bf2d4288c3b86bf008f490a699c806"
server_key_version: "encryptionkey@0"
I20260620 23:29:48.013316 9818 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:29:48.018319 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:48.021147 9818 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:29:48.021536 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "878b3ef70c624729a82e421c36ce5321"
format_stamp: "Formatted at 2026-06-20 23:29:47 on dist-test-slave-v34r"
server_key: "dd331186f8648f3a17f65bbbb36e069c"
server_key_iv: "92bf2d4288c3b86bf008f490a699c806"
server_key_version: "encryptionkey@0"
I20260620 23:29:48.021904 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:48.062064 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:48.063119 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:48.063437 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:48.064561 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:48.066406 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:48.066507 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:48.066592 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:48.066642 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:48.104878 9818 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:42409
I20260620 23:29:48.104972 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:42409 every 8 connection(s)
I20260620 23:29:48.106565 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:48.111972 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9818
I20260620 23:29:48.112185 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:48.112850 387 external_mini_cluster.cc:1428] Setting key f7193bacd24ea5103ddc719199442cb6
I20260620 23:29:48.115706 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39949
--builtin_ntp_servers=127.0.96.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:29:48.121707 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:48.109673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52407 (local address 127.0.96.254:39949)
0620 23:29:48.110214 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:48.110219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:48.111592 (+ 1373us) server_negotiation.cc:407] Connection header received
0620 23:29:48.113941 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:48.113947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:48.114060 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:48.114196 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:48.115574 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.116446 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.118308 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.118774 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.118930 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:48.120651 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:48.120657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:48.120661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:48.120684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:48.120727 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:48.120741 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:48.120843 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:48.121163 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 23:29:48.121286 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":301,"threads_started":1}
I20260620 23:29:48.123032 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:48.110107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39949 (local address 127.0.96.193:52407)
0620 23:29:48.111351 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:48.111406 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:48.113573 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:48.114506 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:48.114517 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:48.114640 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:48.115384 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:48.115402 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.116611 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.116615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:48.118132 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:48.118145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.119045 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.119052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:48.119359 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:48.120237 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:29:48.120239 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:48.120294 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:29:48.120412 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:48.120842 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:48.120863 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:48.120877 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:29:48.121144 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 23:29:48.121677 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":385,"threads_started":1}
I20260620 23:29:48.125389 9945 heartbeater.cc:347] Connected to a master server at 127.0.96.254:39949
I20260620 23:29:48.125808 9945 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:48.126858 9945 heartbeater.cc:511] Master 127.0.96.254:39949 requested a full tablet report, sending...
I20260620 23:29:48.129931 9759 ts_manager.cc:194] Registered new tserver with Master: 878b3ef70c624729a82e421c36ce5321 (127.0.96.193:42409)
I20260620 23:29:48.132294 9759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.193:52407
W20260620 23:29:48.454304 9949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:48.454885 9949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:48.454993 9949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:48.462585 9949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:48.462695 9949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:48.462867 9949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:48.472373 9949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:48.473749 9949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:48.475764 9949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:48.492480 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:48.493052 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:48.493112 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:48.493732 9949 server_base.cc:1034] running on GCE node
I20260620 23:29:48.494524 9949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:48.495600 9949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:48.497076 9949 hybrid_clock.cc:648] HybridClock initialized: now 1781998188496935 us; error 65 us; skew 500 ppm
I20260620 23:29:48.497506 9949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:48.500442 9949 webserver.cc:460] Webserver started at http://127.0.96.194:35113/ using document root <none> and password file <none>
I20260620 23:29:48.501308 9949 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:48.501406 9949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:48.501765 9949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:48.504705 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "769e57b77da948b9aac349bc3aa0df84"
format_stamp: "Formatted at 2026-06-20 23:29:48 on dist-test-slave-v34r"
server_key: "cee2d1bc000263027112253c8da960b5"
server_key_iv: "335ef745ad9293a96e9f5949dd18d0a3"
server_key_version: "encryptionkey@0"
I20260620 23:29:48.505560 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "769e57b77da948b9aac349bc3aa0df84"
format_stamp: "Formatted at 2026-06-20 23:29:48 on dist-test-slave-v34r"
server_key: "cee2d1bc000263027112253c8da960b5"
server_key_iv: "335ef745ad9293a96e9f5949dd18d0a3"
server_key_version: "encryptionkey@0"
I20260620 23:29:48.511207 9949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:29:48.515187 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:48.516839 9949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:29:48.517062 9949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "769e57b77da948b9aac349bc3aa0df84"
format_stamp: "Formatted at 2026-06-20 23:29:48 on dist-test-slave-v34r"
server_key: "cee2d1bc000263027112253c8da960b5"
server_key_iv: "335ef745ad9293a96e9f5949dd18d0a3"
server_key_version: "encryptionkey@0"
I20260620 23:29:48.517309 9949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:48.535702 9949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:48.536598 9949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:48.536911 9949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:48.538015 9949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:48.539563 9949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:48.539690 9949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:48.539844 9949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:48.539937 9949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:48.579660 9949 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:46821
I20260620 23:29:48.579579 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:46821 every 8 connection(s)
I20260620 23:29:48.581112 9949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:48.582273 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 9949
I20260620 23:29:48.582475 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:48.583132 387 external_mini_cluster.cc:1428] Setting key e4c8fb962a2849285b380f16a7834a9f
I20260620 23:29:48.586931 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39949
--builtin_ntp_servers=127.0.96.212:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 23:29:48.596803 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:48.584736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:33959 (local address 127.0.96.254:39949)
0620 23:29:48.584969 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:48.585003 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:48.587462 (+ 2459us) server_negotiation.cc:407] Connection header received
0620 23:29:48.589422 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:48.589427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:48.589523 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:48.589635 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:48.590737 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.591472 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.593030 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.593434 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.593575 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:48.595471 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:48.595482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:48.595491 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:48.595526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:48.595590 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:48.595613 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:48.595757 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:48.596021 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 23:29:48.596186 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:29:48.597478 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:48.585660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39949 (local address 127.0.96.194:33959)
0620 23:29:48.587294 (+ 1634us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:48.587338 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:48.589140 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:48.589878 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:48.589886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:48.589979 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:48.590572 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:48.590588 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.591617 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.591623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:48.592845 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:48.592861 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:48.593664 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:48.593671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:48.593913 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:48.594812 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:29:48.594815 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:48.595005 (+ 190us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:29:48.595125 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:48.595771 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:48.595780 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:48.595787 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:29:48.595957 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:29:48.596274 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":460,"threads_started":1}
I20260620 23:29:48.599133 10076 heartbeater.cc:347] Connected to a master server at 127.0.96.254:39949
I20260620 23:29:48.599630 10076 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:48.600816 10076 heartbeater.cc:511] Master 127.0.96.254:39949 requested a full tablet report, sending...
I20260620 23:29:48.602813 9759 ts_manager.cc:194] Registered new tserver with Master: 769e57b77da948b9aac349bc3aa0df84 (127.0.96.194:46821)
I20260620 23:29:48.603761 9759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.194:33959
W20260620 23:29:49.005183 10079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:49.005584 10079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:49.005753 10079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:49.015269 10079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:49.015430 10079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:49.015622 10079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:49.028188 10079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:39949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:49.030092 10079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:49.032531 10079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:49.050273 10084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:49.050560 10085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:49.050540 10087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:49.051765 10079 server_base.cc:1034] running on GCE node
I20260620 23:29:49.052536 10079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:49.053738 10079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:49.054991 10079 hybrid_clock.cc:648] HybridClock initialized: now 1781998189054923 us; error 59 us; skew 500 ppm
I20260620 23:29:49.055367 10079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:49.061631 10079 webserver.cc:460] Webserver started at http://127.0.96.195:43175/ using document root <none> and password file <none>
I20260620 23:29:49.062438 10079 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:49.062531 10079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:49.062808 10079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:49.065876 10079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "144a1c62446746a9ac136aded1f99970"
format_stamp: "Formatted at 2026-06-20 23:29:49 on dist-test-slave-v34r"
server_key: "da50c73dd9eed701e224c86ca33ae171"
server_key_iv: "2d2c3a8aee336b78a47d32c27e8cdbbd"
server_key_version: "encryptionkey@0"
I20260620 23:29:49.066675 10079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "144a1c62446746a9ac136aded1f99970"
format_stamp: "Formatted at 2026-06-20 23:29:49 on dist-test-slave-v34r"
server_key: "da50c73dd9eed701e224c86ca33ae171"
server_key_iv: "2d2c3a8aee336b78a47d32c27e8cdbbd"
server_key_version: "encryptionkey@0"
I20260620 23:29:49.072490 10079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:29:49.076810 10093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:49.078598 10079 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:49.078761 10079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "144a1c62446746a9ac136aded1f99970"
format_stamp: "Formatted at 2026-06-20 23:29:49 on dist-test-slave-v34r"
server_key: "da50c73dd9eed701e224c86ca33ae171"
server_key_iv: "2d2c3a8aee336b78a47d32c27e8cdbbd"
server_key_version: "encryptionkey@0"
I20260620 23:29:49.079020 10079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:49.096815 10079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:49.097839 10079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:49.098225 10079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:49.099372 10079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:49.100960 10079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:49.101120 10079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 23:29:49.101220 10079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:49.101279 10079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:49.136940 9945 heartbeater.cc:503] Master 127.0.96.254:39949 was elected leader, sending a full tablet report...
I20260620 23:29:49.141985 10079 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:40453
I20260620 23:29:49.142083 10205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:40453 every 8 connection(s)
I20260620 23:29:49.143399 10079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:49.143939 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10079
I20260620 23:29:49.144164 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:49.144971 387 external_mini_cluster.cc:1428] Setting key f07aed17f3c4fd2bc80ee2468910cb5b
I20260620 23:29:49.159201 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:49.146646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:57255 (local address 127.0.96.254:39949)
0620 23:29:49.147280 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:49.147285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:49.149102 (+ 1817us) server_negotiation.cc:407] Connection header received
0620 23:29:49.151096 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:49.151100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:49.151185 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:49.151306 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:49.152456 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.153422 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.155077 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.155738 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.155897 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:49.157719 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:49.157735 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:49.157740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:49.157769 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:49.157852 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:49.157873 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:49.158013 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:49.158577 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 23:29:49.158731 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":358,"threads_started":1}
I20260620 23:29:49.161037 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:49.147547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:39949 (local address 127.0.96.195:57255)
0620 23:29:49.148757 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:49.148845 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:49.150796 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:49.151523 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:49.151531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:49.151624 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:49.152239 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:49.152256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.153598 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.153603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:49.154898 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:49.154911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.155995 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.156000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:49.156217 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:49.157125 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:29:49.157128 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:49.157169 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:29:49.157296 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:49.158161 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:49.158182 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:49.158194 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:29:49.158477 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 23:29:49.159103 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":612,"threads_started":1}
I20260620 23:29:49.163656 10206 heartbeater.cc:347] Connected to a master server at 127.0.96.254:39949
I20260620 23:29:49.164139 10206 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:49.165503 10206 heartbeater.cc:511] Master 127.0.96.254:39949 requested a full tablet report, sending...
I20260620 23:29:49.167291 9759 ts_manager.cc:194] Registered new tserver with Master: 144a1c62446746a9ac136aded1f99970 (127.0.96.195:40453)
I20260620 23:29:49.168557 9759 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.195:57255
I20260620 23:29:49.178179 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:29:49.187765 10208 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:49.185356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.96.254:39949)
0620 23:29:49.185566 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:49.185573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:49.186301 (+ 728us) server_negotiation.cc:407] Connection header received
0620 23:29:49.186545 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:49.186552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:49.186642 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:49.186783 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:49.187469 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35470: BlockingRecv error: recv got EOF from 127.0.0.1:35470 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:29:49.188391 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9818
I20260620 23:29:49.201735 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:49.189693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43080 (local address 127.0.96.193:42409)
0620 23:29:49.190566 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:49.190579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:49.190623 (+ 44us) server_negotiation.cc:407] Connection header received
0620 23:29:49.190782 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:49.190792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:49.191270 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:49.191571 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:49.191936 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.193874 (+ 1938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.196040 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.197358 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.197554 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:49.199165 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:49.199179 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:49.199204 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:49.200319 (+ 1115us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:49.200415 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:49.200475 (+ 60us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:49.200624 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:49.200853 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 23:29:49.201123 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":346,"threads_started":1}
I20260620 23:29:49.203534 9940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:49.295925 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9818
I20260620 23:29:49.314360 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9949
I20260620 23:29:49.323208 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:49.315465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.0.96.194:46821)
0620 23:29:49.316153 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:49.316161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:49.316192 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:29:49.316254 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:49.316260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:49.316545 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:49.316738 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:49.317225 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.318261 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.319470 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.320042 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.320185 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:49.321577 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:49.321586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:49.321600 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:49.322309 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:49.322360 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:49.322380 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:49.322506 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:49.322688 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 23:29:49.322841 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":341,"threads_started":1}
I20260620 23:29:49.324455 10071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:49.416060 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9949
I20260620 23:29:49.433184 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10079
I20260620 23:29:49.443185 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:49.434475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51488 (local address 127.0.96.195:40453)
0620 23:29:49.435250 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:49.435260 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:49.435334 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:29:49.435429 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:49.435437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:49.435723 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:49.435943 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:49.436511 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.437768 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.439144 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:49.439807 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:49.439968 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:49.441412 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:49.441422 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:49.441440 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:49.442182 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:49.442249 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:49.442271 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:49.442409 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:49.442588 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 23:29:49.442779 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":415,"threads_started":1}
I20260620 23:29:49.444954 10201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:49.537911 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10079
I20260620 23:29:49.557221 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 9728
I20260620 23:29:49.558720 9789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:49.676093 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 9728
2026-06-20T23:29:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2807 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T23:29:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:49Z Disabled control of system clock
I20260620 23:29:49.763723 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:46455
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:29:50.187501 10226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:50.187917 10226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:50.188015 10226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:50.196147 10226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:50.196314 10226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:50.196357 10226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:50.196419 10226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:50.196502 10226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:50.206947 10226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:46455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:50.208638 10226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:50.210922 10226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:50.225677 10231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:50.225658 10232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:50.225648 10234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:50.226807 10226 server_base.cc:1034] running on GCE node
I20260620 23:29:50.228060 10226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:50.230289 10226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:50.231617 10226 hybrid_clock.cc:648] HybridClock initialized: now 1781998190231530 us; error 81 us; skew 500 ppm
I20260620 23:29:50.232306 10226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:50.237313 10226 webserver.cc:460] Webserver started at http://127.0.96.254:45287/ using document root <none> and password file <none>
I20260620 23:29:50.238183 10226 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:50.238284 10226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:50.238695 10226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:50.241911 10226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "e177d04077ce4db2b6233424d61ea9d4"
format_stamp: "Formatted at 2026-06-20 23:29:50 on dist-test-slave-v34r"
server_key: "9de51826fdfe38c2d6a582f384afcd1b"
server_key_iv: "6c17e70bcdece30f2e687f797f572809"
server_key_version: "encryptionkey@0"
I20260620 23:29:50.242756 10226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "e177d04077ce4db2b6233424d61ea9d4"
format_stamp: "Formatted at 2026-06-20 23:29:50 on dist-test-slave-v34r"
server_key: "9de51826fdfe38c2d6a582f384afcd1b"
server_key_iv: "6c17e70bcdece30f2e687f797f572809"
server_key_version: "encryptionkey@0"
I20260620 23:29:50.248678 10226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 23:29:50.253247 10240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:50.255116 10226 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:29:50.255364 10226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "e177d04077ce4db2b6233424d61ea9d4"
format_stamp: "Formatted at 2026-06-20 23:29:50 on dist-test-slave-v34r"
server_key: "9de51826fdfe38c2d6a582f384afcd1b"
server_key_iv: "6c17e70bcdece30f2e687f797f572809"
server_key_version: "encryptionkey@0"
I20260620 23:29:50.255678 10226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:50.261967 10226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:50.262833 10226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:50.263147 10226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:50.283715 10226 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:46455
I20260620 23:29:50.283775 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:46455 every 8 connection(s)
I20260620 23:29:50.285395 10226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:50.290567 10292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:50.295208 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10226
I20260620 23:29:50.295492 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:50.296373 387 external_mini_cluster.cc:1428] Setting key b7cf320cd7d412e8fc8fa8d9ae85e731
I20260620 23:29:50.307446 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:50.299163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.0.96.254:46455)
0620 23:29:50.300431 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:50.300446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:50.300726 (+ 280us) server_negotiation.cc:407] Connection header received
0620 23:29:50.303061 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:50.303092 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:50.303766 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:50.304374 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:50.305698 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:50.305704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:50.305717 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:50.305730 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:50.305799 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:50.305815 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:50.306071 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:50.306211 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:29:50.306474 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":426,"threads_started":1}
I20260620 23:29:50.308224 10292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4: Bootstrap starting.
I20260620 23:29:50.312712 10292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:50.314073 10292 log.cc:824] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:50.318140 10292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4: No bootstrap required, opened a new log
I20260620 23:29:50.326995 10292 raft_consensus.cc:348] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e177d04077ce4db2b6233424d61ea9d4" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46455 } }
I20260620 23:29:50.327491 10292 raft_consensus.cc:374] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:50.327576 10292 raft_consensus.cc:729] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e177d04077ce4db2b6233424d61ea9d4, State: Initialized, Role: FOLLOWER
I20260620 23:29:50.328325 10292 consensus_queue.cc:260] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e177d04077ce4db2b6233424d61ea9d4" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46455 } }
I20260620 23:29:50.328655 10292 raft_consensus.cc:388] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:50.328807 10292 raft_consensus.cc:482] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:50.329072 10292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:50.332214 10292 raft_consensus.cc:504] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e177d04077ce4db2b6233424d61ea9d4" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46455 } }
I20260620 23:29:50.332962 10292 leader_election.cc:302] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e177d04077ce4db2b6233424d61ea9d4; no voters:
I20260620 23:29:50.333796 10292 leader_election.cc:288] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:50.334045 10297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:50.335090 10297 raft_consensus.cc:686] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [term 1 LEADER]: Becoming Leader. State: Replica: e177d04077ce4db2b6233424d61ea9d4, State: Running, Role: LEADER
I20260620 23:29:50.335721 10297 consensus_queue.cc:237] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e177d04077ce4db2b6233424d61ea9d4" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46455 } }
I20260620 23:29:50.336763 10292 sys_catalog.cc:564] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:50.339057 10298 sys_catalog.cc:455] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e177d04077ce4db2b6233424d61ea9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e177d04077ce4db2b6233424d61ea9d4" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46455 } } }
I20260620 23:29:50.339927 10299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e177d04077ce4db2b6233424d61ea9d4. Latest consensus state: current_term: 1 leader_uuid: "e177d04077ce4db2b6233424d61ea9d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e177d04077ce4db2b6233424d61ea9d4" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46455 } } }
I20260620 23:29:50.340333 10298 sys_catalog.cc:458] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:50.340533 10299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:50.341756 10304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:50.347431 10304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:50.358644 10304 catalog_manager.cc:1347] Generated new cluster ID: 4c44fdc7084242798709783ed680b379
I20260620 23:29:50.358783 10304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:50.382413 10304 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:50.383018 10304 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:50.392341 10304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e177d04077ce4db2b6233424d61ea9d4: Generated new TSK 0
I20260620 23:29:50.393603 10304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:29:50.403995 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46455
--builtin_ntp_servers=127.0.96.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 23:29:50.778204 10316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:50.778491 10316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:50.778584 10316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:50.785918 10316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:50.786047 10316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:50.786159 10316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:50.795431 10316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:50.796716 10316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:50.798477 10316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:50.812552 10321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:50.812628 10322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:50.812652 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:50.813519 10316 server_base.cc:1034] running on GCE node
I20260620 23:29:50.814244 10316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:50.815303 10316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:50.816541 10316 hybrid_clock.cc:648] HybridClock initialized: now 1781998190816486 us; error 41 us; skew 500 ppm
I20260620 23:29:50.816888 10316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:50.819523 10316 webserver.cc:460] Webserver started at http://127.0.96.193:40683/ using document root <none> and password file <none>
I20260620 23:29:50.820322 10316 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:50.820415 10316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:50.820672 10316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:50.823540 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "c64c44a798d949248b1b47b46844bdb7"
format_stamp: "Formatted at 2026-06-20 23:29:50 on dist-test-slave-v34r"
server_key: "3d18815c3bcc626c903c76114387c956"
server_key_iv: "2bb445092bf431df19576d9c895d45cf"
server_key_version: "encryptionkey@0"
I20260620 23:29:50.824306 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "c64c44a798d949248b1b47b46844bdb7"
format_stamp: "Formatted at 2026-06-20 23:29:50 on dist-test-slave-v34r"
server_key: "3d18815c3bcc626c903c76114387c956"
server_key_iv: "2bb445092bf431df19576d9c895d45cf"
server_key_version: "encryptionkey@0"
I20260620 23:29:50.829849 10316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:29:50.833633 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:50.835613 10316 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:29:50.835806 10316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "c64c44a798d949248b1b47b46844bdb7"
format_stamp: "Formatted at 2026-06-20 23:29:50 on dist-test-slave-v34r"
server_key: "3d18815c3bcc626c903c76114387c956"
server_key_iv: "2bb445092bf431df19576d9c895d45cf"
server_key_version: "encryptionkey@0"
I20260620 23:29:50.836063 10316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:50.842506 10316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:50.843379 10316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:50.843657 10316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:50.844635 10316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:50.846190 10316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:50.846318 10316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:50.846457 10316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:50.846550 10316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:50.885936 10316 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:39527
I20260620 23:29:50.886044 10442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:39527 every 8 connection(s)
I20260620 23:29:50.887387 10316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:50.896812 10445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:50.890882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:53135 (local address 127.0.96.254:46455)
0620 23:29:50.891582 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:50.891588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:50.892825 (+ 1237us) server_negotiation.cc:407] Connection header received
0620 23:29:50.894524 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:50.894529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:50.894602 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:50.894754 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:50.896060 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:50.896065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:50.896069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:50.896088 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:50.896140 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:50.896154 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:50.896278 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:50.896461 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 23:29:50.896523 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":427,"threads_started":1}
I20260620 23:29:50.897514 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10316
I20260620 23:29:50.897753 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:50.897949 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:50.891526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:46455 (local address 127.0.96.193:53135)
0620 23:29:50.892619 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:50.892677 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:50.894250 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:50.894953 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:50.894961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:50.895047 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:50.895746 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:29:50.895748 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:50.895785 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:29:50.895866 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:50.896257 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:50.896264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:50.896270 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:29:50.896506 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 23:29:50.896731 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":339,"threads_started":1}
I20260620 23:29:50.898543 387 external_mini_cluster.cc:1428] Setting key 1732ab7611e64846ba165c3b69ade37c
I20260620 23:29:50.900107 10443 heartbeater.cc:347] Connected to a master server at 127.0.96.254:46455
I20260620 23:29:50.900432 10443 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:50.901178 10443 heartbeater.cc:511] Master 127.0.96.254:46455 requested a full tablet report, sending...
I20260620 23:29:50.902045 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46455
--builtin_ntp_servers=127.0.96.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:29:50.903781 10257 ts_manager.cc:194] Registered new tserver with Master: c64c44a798d949248b1b47b46844bdb7 (127.0.96.193:39527)
W20260620 23:29:51.320497 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:51.320868 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:51.321087 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:51.329512 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:51.329651 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:51.329864 10447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:51.340734 10447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:51.342664 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:51.345134 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:51.361927 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:51.362047 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:51.361979 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:51.363433 10447 server_base.cc:1034] running on GCE node
I20260620 23:29:51.364151 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:51.365269 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:51.366513 10447 hybrid_clock.cc:648] HybridClock initialized: now 1781998191366429 us; error 77 us; skew 500 ppm
I20260620 23:29:51.366900 10447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:51.369695 10447 webserver.cc:460] Webserver started at http://127.0.96.194:38703/ using document root <none> and password file <none>
I20260620 23:29:51.370503 10447 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:51.370599 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:51.370981 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:51.374079 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "5f5530c9778f4971bc5b83c268f6700b"
format_stamp: "Formatted at 2026-06-20 23:29:51 on dist-test-slave-v34r"
server_key: "e89d2ed43914600521a0655c0af62e4b"
server_key_iv: "5e29dfd8d0ef1e104d08da817cf01aa4"
server_key_version: "encryptionkey@0"
I20260620 23:29:51.374914 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "5f5530c9778f4971bc5b83c268f6700b"
format_stamp: "Formatted at 2026-06-20 23:29:51 on dist-test-slave-v34r"
server_key: "e89d2ed43914600521a0655c0af62e4b"
server_key_iv: "5e29dfd8d0ef1e104d08da817cf01aa4"
server_key_version: "encryptionkey@0"
I20260620 23:29:51.380622 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:29:51.384707 10461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:51.386392 10447 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:51.386601 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "5f5530c9778f4971bc5b83c268f6700b"
format_stamp: "Formatted at 2026-06-20 23:29:51 on dist-test-slave-v34r"
server_key: "e89d2ed43914600521a0655c0af62e4b"
server_key_iv: "5e29dfd8d0ef1e104d08da817cf01aa4"
server_key_version: "encryptionkey@0"
I20260620 23:29:51.386843 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:51.393368 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:51.394243 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:51.394538 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:51.395572 10447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:51.397162 10447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:51.397279 10447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:51.397419 10447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:51.397513 10447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:51.436308 10447 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:35169
I20260620 23:29:51.436407 10573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:35169 every 8 connection(s)
I20260620 23:29:51.437896 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:51.441476 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10447
I20260620 23:29:51.441749 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:51.442348 387 external_mini_cluster.cc:1428] Setting key c2b704fe133e4a2f0b8a4f7620dc0461
I20260620 23:29:51.445806 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46455
--builtin_ntp_servers=127.0.96.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 23:29:51.448805 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:51.441336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:43767 (local address 127.0.96.254:46455)
0620 23:29:51.442019 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:51.442027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:51.443708 (+ 1681us) server_negotiation.cc:407] Connection header received
0620 23:29:51.445960 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:51.445966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:51.446053 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:51.446203 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:51.447887 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:51.447894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:51.447899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:51.447921 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:51.447980 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:51.447994 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:51.448144 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:51.448358 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 23:29:51.448451 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":277,"threads_started":1}
I20260620 23:29:51.450109 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:51.442031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:46455 (local address 127.0.96.194:43767)
0620 23:29:51.443514 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:51.443563 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:51.445598 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:51.446525 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:51.446536 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:51.446665 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:51.447517 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:29:51.447520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:51.447571 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:29:51.447683 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:51.448104 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:51.448118 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:51.448126 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:29:51.448468 (+ 342us) client_negotiation.cc:240] Negotiation successful
0620 23:29:51.448830 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":398,"threads_started":1}
I20260620 23:29:51.452042 10574 heartbeater.cc:347] Connected to a master server at 127.0.96.254:46455
I20260620 23:29:51.452528 10574 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:51.453524 10574 heartbeater.cc:511] Master 127.0.96.254:46455 requested a full tablet report, sending...
I20260620 23:29:51.455780 10257 ts_manager.cc:194] Registered new tserver with Master: 5f5530c9778f4971bc5b83c268f6700b (127.0.96.194:35169)
W20260620 23:29:51.863687 10578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:51.864091 10578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:51.864197 10578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:51.872511 10578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:51.872658 10578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:51.872793 10578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:51.883126 10578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:46455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:51.885147 10578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:51.887486 10578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:51.904309 10586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:51.904394 10584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:51.904280 10583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:51.907431 10443 heartbeater.cc:503] Master 127.0.96.254:46455 was elected leader, sending a full tablet report...
I20260620 23:29:51.914958 10578 server_base.cc:1034] running on GCE node
I20260620 23:29:51.915802 10578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:51.917064 10578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:51.918327 10578 hybrid_clock.cc:648] HybridClock initialized: now 1781998191918267 us; error 59 us; skew 500 ppm
I20260620 23:29:51.918735 10578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:29:51.921514 10578 webserver.cc:460] Webserver started at http://127.0.96.195:45107/ using document root <none> and password file <none>
I20260620 23:29:51.922375 10578 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:51.922469 10578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:51.922789 10578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:51.925909 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "4be712c101c24ef59c9cb0e8f4928081"
format_stamp: "Formatted at 2026-06-20 23:29:51 on dist-test-slave-v34r"
server_key: "45056b8394fb77a97c69cc95ca812eae"
server_key_iv: "cea674289b5a3b1b8d5ba31729f8f70e"
server_key_version: "encryptionkey@0"
I20260620 23:29:51.926729 10578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "4be712c101c24ef59c9cb0e8f4928081"
format_stamp: "Formatted at 2026-06-20 23:29:51 on dist-test-slave-v34r"
server_key: "45056b8394fb77a97c69cc95ca812eae"
server_key_iv: "cea674289b5a3b1b8d5ba31729f8f70e"
server_key_version: "encryptionkey@0"
I20260620 23:29:51.932492 10578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:29:51.936941 10592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:51.938993 10578 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:51.939173 10578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "4be712c101c24ef59c9cb0e8f4928081"
format_stamp: "Formatted at 2026-06-20 23:29:51 on dist-test-slave-v34r"
server_key: "45056b8394fb77a97c69cc95ca812eae"
server_key_iv: "cea674289b5a3b1b8d5ba31729f8f70e"
server_key_version: "encryptionkey@0"
I20260620 23:29:51.939399 10578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:51.945976 10578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:51.946811 10578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:51.947036 10578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:51.947955 10578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:51.949471 10578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:51.949570 10578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:51.949666 10578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:51.949724 10578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:51.988147 10578 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:46883
I20260620 23:29:51.988271 10704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:46883 every 8 connection(s)
I20260620 23:29:51.989635 10578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:51.997887 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10578
I20260620 23:29:51.998129 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:51.998678 387 external_mini_cluster.cc:1428] Setting key 6f2f41a9bed15d835643e6bfe0ab0484
I20260620 23:29:52.000681 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:51.993381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:46439 (local address 127.0.96.254:46455)
0620 23:29:51.993986 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:51.993992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:51.995581 (+ 1589us) server_negotiation.cc:407] Connection header received
0620 23:29:51.997537 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:51.997542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:51.997631 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:51.997753 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:51.999562 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:51.999568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:51.999573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:51.999597 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:51.999710 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:51.999738 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:51.999912 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:52.000149 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 23:29:52.000235 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":309,"threads_started":1}
I20260620 23:29:52.001722 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:51.994169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:46455 (local address 127.0.96.195:46439)
0620 23:29:51.995360 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:51.995414 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:51.997244 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:51.998016 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:51.998025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:51.998180 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:51.999113 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:29:51.999115 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:51.999156 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:29:51.999247 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:51.999904 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:51.999913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:51.999920 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:29:52.000232 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 23:29:52.000520 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":331,"threads_started":1}
I20260620 23:29:52.003355 10705 heartbeater.cc:347] Connected to a master server at 127.0.96.254:46455
I20260620 23:29:52.003806 10705 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:52.004606 10705 heartbeater.cc:511] Master 127.0.96.254:46455 requested a full tablet report, sending...
I20260620 23:29:52.006594 10257 ts_manager.cc:194] Registered new tserver with Master: 4be712c101c24ef59c9cb0e8f4928081 (127.0.96.195:46883)
I20260620 23:29:52.015719 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 23:29:52.024555 10707 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:29:52.022249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55278 (local address 127.0.96.254:46455)
0620 23:29:52.022535 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:52.022543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:52.023368 (+ 825us) server_negotiation.cc:407] Connection header received
0620 23:29:52.023592 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:52.023598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:52.023673 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:52.023801 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:52.024288 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55278: BlockingRecv error: recv got EOF from 127.0.0.1:55278 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 23:29:52.025449 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10316
I20260620 23:29:52.032047 10715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:52.026993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.96.193:39527)
0620 23:29:52.029592 (+ 2599us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:52.029601 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:52.029653 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:29:52.029738 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:52.029745 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:52.030046 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:52.030227 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:52.031367 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:52.031374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:52.031386 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:52.031399 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:52.031443 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:52.031457 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:52.031606 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:52.031683 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 23:29:52.031766 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2381,"thread_start_us":2205,"threads_started":1}
I20260620 23:29:52.033345 10438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:52.118749 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10316
I20260620 23:29:52.138936 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10447
I20260620 23:29:52.142921 10717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:52.139941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.0.96.194:35169)
0620 23:29:52.140618 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:52.140626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:52.140655 (+ 29us) server_negotiation.cc:407] Connection header received
0620 23:29:52.140718 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:52.140724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:52.140955 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:52.141184 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:52.142185 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:52.142200 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:52.142212 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:52.142226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:52.142265 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:52.142279 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:52.142427 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:52.142516 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 23:29:52.142617 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":307,"threads_started":1}
I20260620 23:29:52.144639 10569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:52.235879 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10447
I20260620 23:29:52.254130 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10578
I20260620 23:29:52.258610 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:52.255355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.0.96.195:46883)
0620 23:29:52.256066 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:52.256074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:52.256107 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:29:52.256169 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:52.256175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:52.256403 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:52.256674 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:52.257865 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:52.257871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:52.257883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:29:52.257894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:52.257932 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:29:52.257948 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:52.258143 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:52.258207 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 23:29:52.258310 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":343,"threads_started":1}
I20260620 23:29:52.260370 10700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:52.351953 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10578
I20260620 23:29:52.370774 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10226
I20260620 23:29:52.371942 10287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:52.478024 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10226
2026-06-20T23:29:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2797 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:52 dist-test-slave-v34r krb5kdc[10724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:52 dist-test-slave-v34r krb5kdc[10724](info): set up 2 sockets
Jun 20 23:29:52 dist-test-slave-v34r krb5kdc[10724](info): commencing operation
krb5kdc: starting...
W20260620 23:29:53.606024 387 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:53 dist-test-slave-v34r krb5kdc[10724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:29:53.876727 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36935
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:54.235765 10737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:54.236115 10737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:54.236307 10737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:54.243716 10737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:29:54.243872 10737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:54.243978 10737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:54.244082 10737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:29:54.244166 10737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:29:54.256738 10737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:54.258147 10737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:54.259438 10737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:54.274123 10742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:54.274031 10743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:54.274148 10745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:54.275121 10737 server_base.cc:1034] running on GCE node
I20260620 23:29:54.276165 10737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:54.278218 10737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:54.279507 10737 hybrid_clock.cc:648] HybridClock initialized: now 1781998194279461 us; error 70 us; skew 500 ppm
I20260620 23:29:54.279976 10737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:54 dist-test-slave-v34r krb5kdc[10724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:54.284125 10737 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:29:54.286306 10737 webserver.cc:460] Webserver started at http://127.0.96.254:40145/ using document root <none> and password file <none>
I20260620 23:29:54.287240 10737 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:54.287418 10737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:54.287786 10737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:54.291796 10737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "48803e1729994ede80b7ab235ef32ba6"
format_stamp: "Formatted at 2026-06-20 23:29:54 on dist-test-slave-v34r"
server_key: "c9495733e0c54ffc2716801490aa98e1"
server_key_iv: "36133e761d595fca475e5c780760d0b0"
server_key_version: "encryptionkey@0"
I20260620 23:29:54.292877 10737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "48803e1729994ede80b7ab235ef32ba6"
format_stamp: "Formatted at 2026-06-20 23:29:54 on dist-test-slave-v34r"
server_key: "c9495733e0c54ffc2716801490aa98e1"
server_key_iv: "36133e761d595fca475e5c780760d0b0"
server_key_version: "encryptionkey@0"
I20260620 23:29:54.300258 10737 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 23:29:54.305656 10752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:54.307674 10737 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:29:54.307963 10737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "48803e1729994ede80b7ab235ef32ba6"
format_stamp: "Formatted at 2026-06-20 23:29:54 on dist-test-slave-v34r"
server_key: "c9495733e0c54ffc2716801490aa98e1"
server_key_iv: "36133e761d595fca475e5c780760d0b0"
server_key_version: "encryptionkey@0"
I20260620 23:29:54.308245 10737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:54.327888 10737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:54.334050 10737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:54.334435 10737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:54.358388 10737 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36935
I20260620 23:29:54.358428 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36935 every 8 connection(s)
I20260620 23:29:54.360280 10737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:29:54.364573 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:54.366807 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10737
I20260620 23:29:54.367069 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:29:54.367874 387 external_mini_cluster.cc:1428] Setting key e3637d19caef65d60d3caa3eba80b2cb
I20260620 23:29:54.377419 10804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6: Bootstrap starting.
I20260620 23:29:54.380453 10804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:54.381634 10804 log.cc:824] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:29:54 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:54.385190 10804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6: No bootstrap required, opened a new log
I20260620 23:29:54.393527 10804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48803e1729994ede80b7ab235ef32ba6" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36935 } }
I20260620 23:29:54.393950 10804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:54.394079 10804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48803e1729994ede80b7ab235ef32ba6, State: Initialized, Role: FOLLOWER
I20260620 23:29:54.394637 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:54.370392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.0.96.254:36935)
0620 23:29:54.371434 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:54.371451 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:54.371507 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:29:54.373548 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:54.373577 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:54.374118 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:54.374635 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:54.376396 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:54.378044 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:54.379465 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:54.380241 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:54.380476 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:54.385779 (+ 5303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:54.385790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:54.385812 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:54.385862 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:54.389473 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:54.390379 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:54.390387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:54.390400 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:54.390619 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:54.391082 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:54.391088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:54.391098 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:54.391797 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:54.392220 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:54.392688 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 23:29:54.393241 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":410,"threads_started":1}
I20260620 23:29:54.394793 10804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48803e1729994ede80b7ab235ef32ba6" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36935 } }
I20260620 23:29:54.395112 10804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:29:54.395267 10804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:29:54.395447 10804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:54.398114 10804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48803e1729994ede80b7ab235ef32ba6" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36935 } }
I20260620 23:29:54.398626 10804 leader_election.cc:302] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48803e1729994ede80b7ab235ef32ba6; no voters:
I20260620 23:29:54.399246 10804 leader_election.cc:288] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:29:54.399457 10809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:54.400298 10809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [term 1 LEADER]: Becoming Leader. State: Replica: 48803e1729994ede80b7ab235ef32ba6, State: Running, Role: LEADER
I20260620 23:29:54.401087 10809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48803e1729994ede80b7ab235ef32ba6" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36935 } }
I20260620 23:29:54.401520 10804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:29:54.404798 10811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48803e1729994ede80b7ab235ef32ba6. Latest consensus state: current_term: 1 leader_uuid: "48803e1729994ede80b7ab235ef32ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48803e1729994ede80b7ab235ef32ba6" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36935 } } }
I20260620 23:29:54.405289 10811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:54.405966 10810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48803e1729994ede80b7ab235ef32ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48803e1729994ede80b7ab235ef32ba6" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36935 } } }
I20260620 23:29:54.406342 10810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6 [sys.catalog]: This master's current role is: LEADER
I20260620 23:29:54.410246 10815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:29:54.417743 10815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:29:54.430538 10815 catalog_manager.cc:1347] Generated new cluster ID: b513b39c36234aa887d019da73074c3f
I20260620 23:29:54.430696 10815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:29:54.453112 10815 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:29:54.454538 10815 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:29:54.464666 10815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6: Generated new TSK 0
I20260620 23:29:54.465731 10815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:29:54.522404 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36935
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:54.943112 10830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:54.943506 10830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:54.943614 10830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:54.951784 10830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:54.951954 10830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:54.952090 10830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:29:54.963587 10830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:54.965497 10830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:54.967236 10830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:54.984238 10836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:54.984328 10838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:54.984189 10835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:54.986742 10830 server_base.cc:1034] running on GCE node
I20260620 23:29:54.987622 10830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:54.988893 10830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:54.990219 10830 hybrid_clock.cc:648] HybridClock initialized: now 1781998194990158 us; error 48 us; skew 500 ppm
I20260620 23:29:54.990640 10830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:54 dist-test-slave-v34r krb5kdc[10724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:54.994743 10830 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:29:54.996850 10830 webserver.cc:460] Webserver started at http://127.0.96.193:44877/ using document root <none> and password file <none>
I20260620 23:29:54.997840 10830 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:54.998004 10830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:54.998375 10830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:55.002218 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "a08e25b4d2724bceb3cc5e9049f0a51a"
format_stamp: "Formatted at 2026-06-20 23:29:54 on dist-test-slave-v34r"
server_key: "a020644c16fe60574f54a6346c7e32e8"
server_key_iv: "14f24c9814c2063ebd8b488213f7f4e5"
server_key_version: "encryptionkey@0"
I20260620 23:29:55.003183 10830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "a08e25b4d2724bceb3cc5e9049f0a51a"
format_stamp: "Formatted at 2026-06-20 23:29:54 on dist-test-slave-v34r"
server_key: "a020644c16fe60574f54a6346c7e32e8"
server_key_iv: "14f24c9814c2063ebd8b488213f7f4e5"
server_key_version: "encryptionkey@0"
I20260620 23:29:55.010426 10830 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260620 23:29:55.016332 10845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:55.018525 10830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:29:55.018779 10830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "a08e25b4d2724bceb3cc5e9049f0a51a"
format_stamp: "Formatted at 2026-06-20 23:29:54 on dist-test-slave-v34r"
server_key: "a020644c16fe60574f54a6346c7e32e8"
server_key_iv: "14f24c9814c2063ebd8b488213f7f4e5"
server_key_version: "encryptionkey@0"
I20260620 23:29:55.019075 10830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:55.048167 10830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:55.054138 10830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:55.054535 10830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:55.055771 10830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:55.057616 10830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:55.057732 10830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:55.057899 10830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:55.057986 10830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:55.098553 10830 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:41203
I20260620 23:29:55.098603 10957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:41203 every 8 connection(s)
I20260620 23:29:55.100175 10830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:29:55.104516 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10830
I20260620 23:29:55.104841 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:29:55.105784 387 external_mini_cluster.cc:1428] Setting key 8a0a4e663cd44a7d657e8c1e465418c2
Jun 20 23:29:55 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:55.126515 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:55.103508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:58811 (local address 127.0.96.254:36935)
0620 23:29:55.104045 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:55.104049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:55.105596 (+ 1547us) server_negotiation.cc:407] Connection header received
0620 23:29:55.108134 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:55.108139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:55.108266 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:55.108406 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:55.110164 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.111506 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.113631 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.114458 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.114656 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:55.119096 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:55.119103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:55.119108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:55.119158 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:55.122106 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:55.123087 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:55.123091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:55.123094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:55.123234 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:55.123847 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:55.123857 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:55.123863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:55.124316 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:55.124486 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:55.125721 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0620 23:29:55.126019 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":347,"threads_started":1}
I20260620 23:29:55.127584 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:55.104325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36935 (local address 127.0.96.193:58811)
0620 23:29:55.105412 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:55.105466 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:55.107787 (+ 2321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:55.108679 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:55.108688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:55.109322 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:55.109946 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:55.109963 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.111856 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.111860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:55.113441 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:55.113455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.114745 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.114750 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:55.114968 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:55.115806 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:55.115830 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:55.118722 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:55.122285 (+ 3563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:55.122297 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:55.122317 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:55.122937 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:55.123387 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:55.123391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:55.123394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:55.123655 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:55.124521 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:55.124532 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:55.125178 (+ 646us) client_negotiation.cc:769] Sending connection context
0620 23:29:55.125547 (+ 369us) client_negotiation.cc:240] Negotiation successful
0620 23:29:55.126091 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":469,"threads_started":1}
I20260620 23:29:55.129678 10958 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36935
I20260620 23:29:55.130085 10958 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:55.131124 10958 heartbeater.cc:511] Master 127.0.96.254:36935 requested a full tablet report, sending...
I20260620 23:29:55.134006 10769 ts_manager.cc:194] Registered new tserver with Master: a08e25b4d2724bceb3cc5e9049f0a51a (127.0.96.193:41203)
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:29:55.135803 10769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:58811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:29:55.168043 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36935
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:55.518144 10964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:55.518471 10964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:55.518639 10964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:29:55.525985 10964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:55.526147 10964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:55.526319 10964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:29:55.536103 10964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:55.537561 10964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:55.538851 10964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:55.552561 10970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:55.552649 10972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:55.552624 10969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:55.553733 10964 server_base.cc:1034] running on GCE node
I20260620 23:29:55.554526 10964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:55.555768 10964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:55.557080 10964 hybrid_clock.cc:648] HybridClock initialized: now 1781998195556947 us; error 51 us; skew 500 ppm
I20260620 23:29:55.557488 10964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:55 dist-test-slave-v34r krb5kdc[10724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:55.561379 10964 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:29:55.563352 10964 webserver.cc:460] Webserver started at http://127.0.96.194:46249/ using document root <none> and password file <none>
I20260620 23:29:55.564250 10964 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:55.564391 10964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:55.564759 10964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:55.568379 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "2e14e88c5ce44358bb34ea06bda4f054"
format_stamp: "Formatted at 2026-06-20 23:29:55 on dist-test-slave-v34r"
server_key: "b4144406f178f7a231ef8445037a1b54"
server_key_iv: "d51604749d19b37c5342a0670ab5f4f7"
server_key_version: "encryptionkey@0"
I20260620 23:29:55.569411 10964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "2e14e88c5ce44358bb34ea06bda4f054"
format_stamp: "Formatted at 2026-06-20 23:29:55 on dist-test-slave-v34r"
server_key: "b4144406f178f7a231ef8445037a1b54"
server_key_iv: "d51604749d19b37c5342a0670ab5f4f7"
server_key_version: "encryptionkey@0"
I20260620 23:29:55.576166 10964 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260620 23:29:55.580793 10979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:55.582756 10964 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:55.583009 10964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "2e14e88c5ce44358bb34ea06bda4f054"
format_stamp: "Formatted at 2026-06-20 23:29:55 on dist-test-slave-v34r"
server_key: "b4144406f178f7a231ef8445037a1b54"
server_key_iv: "d51604749d19b37c5342a0670ab5f4f7"
server_key_version: "encryptionkey@0"
I20260620 23:29:55.583281 10964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:55.600291 10964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:55.605532 10964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:55.605886 10964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:55.607064 10964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:55.608778 10964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:55.608862 10964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:55.609086 10964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:55.609148 10964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:55.648828 10964 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:42399
I20260620 23:29:55.648932 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:42399 every 8 connection(s)
I20260620 23:29:55.650466 10964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:29:55.657821 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 10964
I20260620 23:29:55.658179 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:29:55.658696 387 external_mini_cluster.cc:1428] Setting key 9e3e6e2cdb52dd881bc5ae6f2950317e
Jun 20 23:29:55 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:55.675534 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:55.653591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:55975 (local address 127.0.96.254:36935)
0620 23:29:55.654067 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:55.654072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:55.655311 (+ 1239us) server_negotiation.cc:407] Connection header received
0620 23:29:55.657269 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:55.657275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:55.657389 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:55.657508 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:55.659273 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.660135 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.662201 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.663012 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.663171 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:55.667637 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:55.667646 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:55.667654 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:55.667701 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:55.671457 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:55.672441 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:55.672450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:55.672457 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:55.672688 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:55.673240 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:55.673248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:55.673254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:55.673689 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:55.673872 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:55.674674 (+ 802us) server_negotiation.cc:299] Negotiation successful
0620 23:29:55.674988 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":268,"threads_started":1}
I20260620 23:29:55.676146 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:55.654208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36935 (local address 127.0.96.194:55975)
0620 23:29:55.655132 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:55.655176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:55.656967 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:55.657755 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:55.657764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:55.658450 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:55.659100 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:55.659119 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.660343 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.660349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:55.662005 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:55.662023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:55.663323 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:55.663329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:55.663607 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:55.664441 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:55.664467 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:55.667332 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:55.671701 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:55.671710 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:55.671725 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:55.672255 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:55.672817 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:55.672821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:55.672823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:55.673102 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:55.673856 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:55.673865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:55.674256 (+ 391us) client_negotiation.cc:769] Sending connection context
0620 23:29:55.674496 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 23:29:55.674902 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":380,"threads_started":1}
I20260620 23:29:55.678078 11092 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36935
I20260620 23:29:55.678413 11092 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:55.679214 11092 heartbeater.cc:511] Master 127.0.96.254:36935 requested a full tablet report, sending...
I20260620 23:29:55.680846 10769 ts_manager.cc:194] Registered new tserver with Master: 2e14e88c5ce44358bb34ea06bda4f054 (127.0.96.194:42399)
I20260620 23:29:55.681725 10769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:55975
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:29:55.721218 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36935
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:33213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:29:56.135171 11098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:29:56.135650 11098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:29:56.135833 11098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 23:29:56.139741 10958 heartbeater.cc:503] Master 127.0.96.254:36935 was elected leader, sending a full tablet report...
W20260620 23:29:56.144608 11098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:29:56.144820 11098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:29:56.145063 11098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:29:56.156765 11098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:33213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:29:56.158633 11098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:29:56.160324 11098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:29:56.177132 11103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:56.177623 11104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:29:56.178491 11106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:29:56.179311 11098 server_base.cc:1034] running on GCE node
I20260620 23:29:56.180382 11098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:29:56.181715 11098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:29:56.183009 11098 hybrid_clock.cc:648] HybridClock initialized: now 1781998196182947 us; error 43 us; skew 500 ppm
I20260620 23:29:56.183454 11098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:29:56 dist-test-slave-v34r krb5kdc[10724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:29:56.187582 11098 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:29:56.189878 11098 webserver.cc:460] Webserver started at http://127.0.96.195:43855/ using document root <none> and password file <none>
I20260620 23:29:56.190845 11098 fs_manager.cc:359] Metadata directory not provided
I20260620 23:29:56.191020 11098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:29:56.191390 11098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:29:56.195190 11098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "3ef405c6cc244f2586a3880dbb3f85fa"
format_stamp: "Formatted at 2026-06-20 23:29:56 on dist-test-slave-v34r"
server_key: "d008c87c351fa7c8ed9ff4481fab1978"
server_key_iv: "19cd2dfeae79ea61d85f320841ad0368"
server_key_version: "encryptionkey@0"
I20260620 23:29:56.196205 11098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "3ef405c6cc244f2586a3880dbb3f85fa"
format_stamp: "Formatted at 2026-06-20 23:29:56 on dist-test-slave-v34r"
server_key: "d008c87c351fa7c8ed9ff4481fab1978"
server_key_iv: "19cd2dfeae79ea61d85f320841ad0368"
server_key_version: "encryptionkey@0"
I20260620 23:29:56.203574 11098 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260620 23:29:56.208712 11113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:56.210804 11098 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:29:56.211105 11098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "3ef405c6cc244f2586a3880dbb3f85fa"
format_stamp: "Formatted at 2026-06-20 23:29:56 on dist-test-slave-v34r"
server_key: "d008c87c351fa7c8ed9ff4481fab1978"
server_key_iv: "19cd2dfeae79ea61d85f320841ad0368"
server_key_version: "encryptionkey@0"
I20260620 23:29:56.211411 11098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:29:56.231601 11098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:29:56.236195 11098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:29:56.236541 11098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:29:56.237677 11098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:29:56.239244 11098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:29:56.239351 11098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:56.239512 11098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:29:56.239593 11098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:29:56.279076 11098 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:37085
I20260620 23:29:56.279170 11225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:37085 every 8 connection(s)
I20260620 23:29:56.280424 11098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:29:56.290086 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 11098
I20260620 23:29:56.290323 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:29:56.290858 387 external_mini_cluster.cc:1428] Setting key fa22e2561f358de2c7b5de6235813352
Jun 20 23:29:56 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:29:56.305179 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.283637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:48183 (local address 127.0.96.254:36935)
0620 23:29:56.284171 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.284176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.285245 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 23:29:56.287455 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.287459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.287535 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.287694 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:56.289761 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.290723 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.293127 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.293758 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.293901 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.298344 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:56.298381 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:56.298385 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:56.298421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:56.301708 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.302700 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.302704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.302707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.302873 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.303352 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.303356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.303358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.303662 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:56.303793 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.304517 (+ 724us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.304731 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":254,"threads_started":1}
I20260620 23:29:56.306241 11229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.284125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36935 (local address 127.0.96.195:48183)
0620 23:29:56.284944 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:56.285087 (+ 143us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:56.287027 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:56.287971 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:56.287981 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:56.288719 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:29:56.289527 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.289547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.290912 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.290920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.292927 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.292946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.294041 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.294047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.294354 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.295279 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:29:56.295307 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:29:56.298039 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:29:56.301902 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:56.301912 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:56.301928 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:56.302537 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:56.303015 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:29:56.303019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:29:56.303022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:29:56.303216 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:29:56.303784 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:29:56.303790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:29:56.304148 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 23:29:56.304395 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 23:29:56.304871 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":298,"threads_started":1}
I20260620 23:29:56.307781 11226 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36935
I20260620 23:29:56.308096 11226 heartbeater.cc:464] Registering TS with master...
I20260620 23:29:56.308810 11226 heartbeater.cc:511] Master 127.0.96.254:36935 requested a full tablet report, sending...
I20260620 23:29:56.310348 10769 ts_manager.cc:194] Registered new tserver with Master: 3ef405c6cc244f2586a3880dbb3f85fa (127.0.96.195:37085)
I20260620 23:29:56.311316 10769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:48183
I20260620 23:29:56.320490 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:29:56.340003 11228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.326123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.0.96.254:36935)
0620 23:29:56.326423 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.326431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.326865 (+ 434us) server_negotiation.cc:407] Connection header received
0620 23:29:56.327039 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.327043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.327120 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.327217 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:56.328760 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.329663 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.330986 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.331421 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.331529 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.334292 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:56.334297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:56.334301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:56.334334 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:56.337020 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.337875 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.337879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.337881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.338001 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.338389 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.338393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.338395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.338799 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:56.339001 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.339394 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.339616 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 23:29:56.345357 10769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:29:56.353345 10769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:29:56.383199 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.371841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58668 (local address 127.0.96.193:41203)
0620 23:29:56.374642 (+ 2801us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.374650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.374680 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:29:56.374782 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.374792 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.375053 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.376019 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:56.376238 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.377840 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.380788 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.381941 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.382101 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.382360 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.382671 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.382837 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2596,"thread_start_us":2526,"threads_started":1}
I20260620 23:29:56.383229 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.370463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41203 (local address 127.0.0.1:58668)
0620 23:29:56.373912 (+ 3449us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:56.373943 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:56.374052 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:56.375301 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:56.375308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:56.375338 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:56.375782 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.375794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.377993 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.378000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.380551 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.380581 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.382203 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.382210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.382419 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.382474 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 23:29:56.382631 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:29:56.382837 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3314,"thread_start_us":6573,"threads_started":1}
I20260620 23:29:56.385409 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.370028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37085 (local address 127.0.0.1:43012)
0620 23:29:56.371275 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:56.371328 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:56.371527 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:56.375881 (+ 4354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:56.375886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:56.375914 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:56.376604 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.376622 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.379572 (+ 2950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.379577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.382087 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.382103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.384540 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.384548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.384749 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.384795 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:29:56.384893 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:29:56.385117 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":1687,"threads_started":1}
I20260620 23:29:56.386385 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.370973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.0.96.195:37085)
0620 23:29:56.374765 (+ 3792us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.374776 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.374817 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:29:56.374950 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.374959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.375287 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.375548 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:56.377766 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.379348 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.382309 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.383909 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.384087 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.384352 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.385860 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.386033 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3566,"thread_start_us":3621,"threads_started":1}
I20260620 23:29:56.387463 10893 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3b0fa315214372bc0e3bdb0e6d7827 (DEFAULT_TABLE table=test-table [id=3826e032dd28430aa238e3adffef9382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:56.389250 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.370919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:42399 (local address 127.0.0.1:49146)
0620 23:29:56.377397 (+ 6478us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:56.377420 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:56.377587 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:56.380153 (+ 2566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:56.380158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:56.380185 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:56.380728 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.380741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.382777 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.382781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.386075 (+ 3294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.386090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.388159 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.388166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.388343 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.388377 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:29:56.388487 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:29:56.388625 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6338,"mutex_wait_us":1,"thread_start_us":2721,"threads_started":1}
I20260620 23:29:56.389276 10893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3b0fa315214372bc0e3bdb0e6d7827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:56.389930 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.371472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49146 (local address 127.0.96.194:42399)
0620 23:29:56.373484 (+ 2012us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.373493 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.377572 (+ 4079us) server_negotiation.cc:407] Connection header received
0620 23:29:56.377751 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.377761 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.378121 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.378364 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:56.381004 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.382593 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.386287 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.387906 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.388071 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.388270 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.388568 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.389612 (+ 1044us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1845,"thread_start_us":1707,"threads_started":1}
I20260620 23:29:56.390857 11161 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3b0fa315214372bc0e3bdb0e6d7827 (DEFAULT_TABLE table=test-table [id=3826e032dd28430aa238e3adffef9382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:56.392527 11161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3b0fa315214372bc0e3bdb0e6d7827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:56.395561 11027 tablet_service.cc:1467] Processing CreateTablet for tablet 5f3b0fa315214372bc0e3bdb0e6d7827 (DEFAULT_TABLE table=test-table [id=3826e032dd28430aa238e3adffef9382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:29:56.396857 11027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f3b0fa315214372bc0e3bdb0e6d7827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:29:56.405423 11245 tablet_bootstrap.cc:492] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Bootstrap starting.
I20260620 23:29:56.409967 11246 tablet_bootstrap.cc:492] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Bootstrap starting.
I20260620 23:29:56.410693 11245 tablet_bootstrap.cc:654] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:56.411953 11247 tablet_bootstrap.cc:492] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Bootstrap starting.
I20260620 23:29:56.413451 11245 log.cc:824] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:56.414276 11246 tablet_bootstrap.cc:654] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:56.415445 11246 log.cc:824] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:56.416664 11247 tablet_bootstrap.cc:654] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Neither blocks nor log segments found. Creating new log.
I20260620 23:29:56.417354 11245 tablet_bootstrap.cc:492] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: No bootstrap required, opened a new log
I20260620 23:29:56.417654 11245 ts_tablet_manager.cc:1397] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260620 23:29:56.418195 11247 log.cc:824] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Log is configured to *not* fsync() on all Append() calls
I20260620 23:29:56.418769 11246 tablet_bootstrap.cc:492] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: No bootstrap required, opened a new log
I20260620 23:29:56.419142 11246 ts_tablet_manager.cc:1397] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260620 23:29:56.421453 11247 tablet_bootstrap.cc:492] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: No bootstrap required, opened a new log
I20260620 23:29:56.421768 11247 ts_tablet_manager.cc:1397] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 23:29:56.429039 11245 raft_consensus.cc:348] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.429540 11245 raft_consensus.cc:374] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:56.429695 11245 raft_consensus.cc:729] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08e25b4d2724bceb3cc5e9049f0a51a, State: Initialized, Role: FOLLOWER
I20260620 23:29:56.429864 11246 raft_consensus.cc:348] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.430296 11246 raft_consensus.cc:374] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:56.430397 11246 raft_consensus.cc:729] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef405c6cc244f2586a3880dbb3f85fa, State: Initialized, Role: FOLLOWER
I20260620 23:29:56.430435 11245 consensus_queue.cc:260] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.430316 11247 raft_consensus.cc:348] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.430968 11247 raft_consensus.cc:374] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:29:56.431137 11247 raft_consensus.cc:729] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e14e88c5ce44358bb34ea06bda4f054, State: Initialized, Role: FOLLOWER
I20260620 23:29:56.430984 11246 consensus_queue.cc:260] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.432014 11247 consensus_queue.cc:260] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.432462 11245 ts_tablet_manager.cc:1428] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260620 23:29:56.433458 11092 heartbeater.cc:503] Master 127.0.96.254:36935 was elected leader, sending a full tablet report...
I20260620 23:29:56.433982 11247 ts_tablet_manager.cc:1428] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260620 23:29:56.435022 11226 heartbeater.cc:503] Master 127.0.96.254:36935 was elected leader, sending a full tablet report...
I20260620 23:29:56.435659 11246 ts_tablet_manager.cc:1428] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Time spent starting tablet: real 0.016s user 0.009s sys 0.005s
W20260620 23:29:56.532325 11227 tablet.cc:2369] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:56.601076 11253 raft_consensus.cc:482] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:29:56.601421 11253 raft_consensus.cc:504] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.603461 11253 leader_election.cc:288] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ef405c6cc244f2586a3880dbb3f85fa (127.0.96.195:37085), a08e25b4d2724bceb3cc5e9049f0a51a (127.0.96.193:41203)
W20260620 23:29:56.603688 10959 tablet.cc:2369] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:56.611667 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.604100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:60359 (local address 127.0.96.193:41203)
0620 23:29:56.604393 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.604404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.605680 (+ 1276us) server_negotiation.cc:407] Connection header received
0620 23:29:56.605927 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.605934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.606013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.606162 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:56.606877 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.607660 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.609730 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.610514 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.610727 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.610817 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.611258 (+ 441us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.611396 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 23:29:56.611760 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.604447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:36289 (local address 127.0.96.195:37085)
0620 23:29:56.605018 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.605028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.605372 (+ 344us) server_negotiation.cc:407] Connection header received
0620 23:29:56.605630 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.605640 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.605736 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.605853 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:29:56.606580 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.607471 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.609698 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.610517 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.610725 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.610826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.611334 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.611479 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260620 23:29:56.611891 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.604037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:41203 (local address 127.0.96.194:60359)
0620 23:29:56.605502 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:56.605543 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:56.605717 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:56.606190 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:56.606193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:56.606214 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:56.606715 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.606726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.607811 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.607815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.609542 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.609555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.610816 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.610823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.611072 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.611121 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 23:29:56.611204 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 23:29:56.611396 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":7,"thread_start_us":989,"threads_started":1}
I20260620 23:29:56.612097 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.603901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37085 (local address 127.0.96.194:36289)
0620 23:29:56.605170 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0620 23:29:56.605198 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:29:56.605445 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:29:56.605924 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:29:56.605930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:29:56.605962 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:29:56.606412 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.606427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.607627 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.607633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.609533 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:29:56.609545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.610859 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.610869 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:29:56.611132 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.611162 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:29:56.611286 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 23:29:56.611468 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":169,"thread_start_us":697,"threads_started":1}
I20260620 23:29:56.613101 10913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b0fa315214372bc0e3bdb0e6d7827" candidate_uuid: "2e14e88c5ce44358bb34ea06bda4f054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" is_pre_election: true
I20260620 23:29:56.613289 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b0fa315214372bc0e3bdb0e6d7827" candidate_uuid: "2e14e88c5ce44358bb34ea06bda4f054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" is_pre_election: true
I20260620 23:29:56.613749 10913 raft_consensus.cc:2454] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e14e88c5ce44358bb34ea06bda4f054 in term 0.
I20260620 23:29:56.613967 11181 raft_consensus.cc:2454] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e14e88c5ce44358bb34ea06bda4f054 in term 0.
I20260620 23:29:56.614549 10981 leader_election.cc:302] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e14e88c5ce44358bb34ea06bda4f054, a08e25b4d2724bceb3cc5e9049f0a51a; no voters:
I20260620 23:29:56.615005 11253 raft_consensus.cc:2781] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:29:56.615175 11253 raft_consensus.cc:482] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:29:56.615288 11253 raft_consensus.cc:3037] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:56.617673 11253 raft_consensus.cc:504] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.618276 11253 leader_election.cc:288] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [CANDIDATE]: Term 1 election: Requested vote from peers 3ef405c6cc244f2586a3880dbb3f85fa (127.0.96.195:37085), a08e25b4d2724bceb3cc5e9049f0a51a (127.0.96.193:41203)
I20260620 23:29:56.618824 10913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b0fa315214372bc0e3bdb0e6d7827" candidate_uuid: "2e14e88c5ce44358bb34ea06bda4f054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a"
I20260620 23:29:56.618897 11181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f3b0fa315214372bc0e3bdb0e6d7827" candidate_uuid: "2e14e88c5ce44358bb34ea06bda4f054" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef405c6cc244f2586a3880dbb3f85fa"
I20260620 23:29:56.619035 10913 raft_consensus.cc:3037] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:56.619216 11181 raft_consensus.cc:3037] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:29:56.621435 10913 raft_consensus.cc:2454] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e14e88c5ce44358bb34ea06bda4f054 in term 1.
I20260620 23:29:56.621961 10981 leader_election.cc:302] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e14e88c5ce44358bb34ea06bda4f054, a08e25b4d2724bceb3cc5e9049f0a51a; no voters:
I20260620 23:29:56.622354 11253 raft_consensus.cc:2781] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:29:56.622318 11181 raft_consensus.cc:2454] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e14e88c5ce44358bb34ea06bda4f054 in term 1.
I20260620 23:29:56.622995 11253 raft_consensus.cc:686] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 1 LEADER]: Becoming Leader. State: Replica: 2e14e88c5ce44358bb34ea06bda4f054, State: Running, Role: LEADER
I20260620 23:29:56.623535 11253 consensus_queue.cc:237] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } }
I20260620 23:29:56.628582 10769 catalog_manager.cc:5495] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e14e88c5ce44358bb34ea06bda4f054 (127.0.96.194). New cstate: current_term: 1 leader_uuid: "2e14e88c5ce44358bb34ea06bda4f054" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e14e88c5ce44358bb34ea06bda4f054" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 42399 } health_report { overall_health: HEALTHY } } }
W20260620 23:29:56.653079 11093 tablet.cc:2369] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:29:56.687289 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.680829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49150 (local address 127.0.96.194:42399)
0620 23:29:56.681167 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.681172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.681237 (+ 65us) server_negotiation.cc:407] Connection header received
0620 23:29:56.681327 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.681332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.681410 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.681537 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:29:56.682068 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.682948 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.684611 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.685141 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.685291 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.685883 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:29:56.686613 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:29:56.686767 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.686914 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.687008 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 23:29:56.697337 10913 raft_consensus.cc:1261] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 1 FOLLOWER]: Refusing update from remote peer 2e14e88c5ce44358bb34ea06bda4f054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:56.697557 11181 raft_consensus.cc:1261] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 1 FOLLOWER]: Refusing update from remote peer 2e14e88c5ce44358bb34ea06bda4f054: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:29:56.698669 11256 consensus_queue.cc:1035] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a08e25b4d2724bceb3cc5e9049f0a51a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 41203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:56.699200 11253 consensus_queue.cc:1035] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef405c6cc244f2586a3880dbb3f85fa" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:29:56.734572 10769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:56.734941 10769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:29:56.740003 10769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48803e1729994ede80b7ab235ef32ba6: Sending DeleteTablet for 3 replicas of tablet 5f3b0fa315214372bc0e3bdb0e6d7827
I20260620 23:29:56.741572 11161 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3b0fa315214372bc0e3bdb0e6d7827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:56 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:43012
I20260620 23:29:56.741694 10893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3b0fa315214372bc0e3bdb0e6d7827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:56 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:58668
I20260620 23:29:56.741701 11027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f3b0fa315214372bc0e3bdb0e6d7827 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:29:56 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:49146
I20260620 23:29:56.746901 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10830
I20260620 23:29:56.754025 11271 tablet_replica.cc:330] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: stopping tablet replica
I20260620 23:29:56.754510 11270 tablet_replica.cc:330] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: stopping tablet replica
I20260620 23:29:56.754787 11271 raft_consensus.cc:2229] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:56.755471 11270 raft_consensus.cc:2229] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:29:56.755594 11271 raft_consensus.cc:2258] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:56.756152 11269 tablet_replica.cc:330] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: stopping tablet replica
I20260620 23:29:56.756501 11270 raft_consensus.cc:2258] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:56.756683 11269 raft_consensus.cc:2229] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:29:56.757256 11269 raft_consensus.cc:2258] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:29:56.758773 11271 ts_tablet_manager.cc:1905] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:56.759644 11270 ts_tablet_manager.cc:1905] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:56.766499 11271 ts_tablet_manager.cc:1918] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:56.768362 11269 ts_tablet_manager.cc:1905] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:29:56.769122 11270 ts_tablet_manager.cc:1918] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:56.769531 11270 log.cc:1196] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/5f3b0fa315214372bc0e3bdb0e6d7827
I20260620 23:29:56.770378 11270 ts_tablet_manager.cc:1939] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 2e14e88c5ce44358bb34ea06bda4f054: Deleting consensus metadata
Jun 20 23:29:56 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:29:56.773736 10756 catalog_manager.cc:4841] TS 2e14e88c5ce44358bb34ea06bda4f054 (127.0.96.194:42399): tablet 5f3b0fa315214372bc0e3bdb0e6d7827 (table test-table [id=3826e032dd28430aa238e3adffef9382]) successfully deleted
I20260620 23:29:56.774647 11271 log.cc:1196] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/5f3b0fa315214372bc0e3bdb0e6d7827
I20260620 23:29:56.775475 11271 ts_tablet_manager.cc:1939] T 5f3b0fa315214372bc0e3bdb0e6d7827 P a08e25b4d2724bceb3cc5e9049f0a51a: Deleting consensus metadata
I20260620 23:29:56.776015 11269 ts_tablet_manager.cc:1918] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:29:56.776345 11269 log.cc:1196] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/5f3b0fa315214372bc0e3bdb0e6d7827
I20260620 23:29:56.777014 11269 ts_tablet_manager.cc:1939] T 5f3b0fa315214372bc0e3bdb0e6d7827 P 3ef405c6cc244f2586a3880dbb3f85fa: Deleting consensus metadata
I20260620 23:29:56.778262 10754 catalog_manager.cc:4841] TS a08e25b4d2724bceb3cc5e9049f0a51a (127.0.96.193:41203): tablet 5f3b0fa315214372bc0e3bdb0e6d7827 (table test-table [id=3826e032dd28430aa238e3adffef9382]) successfully deleted
I20260620 23:29:56.779130 10755 catalog_manager.cc:4841] TS 3ef405c6cc244f2586a3880dbb3f85fa (127.0.96.195:37085): tablet 5f3b0fa315214372bc0e3bdb0e6d7827 (table test-table [id=3826e032dd28430aa238e3adffef9382]) successfully deleted
I20260620 23:29:56.782169 11244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.749095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.0.96.193:41203)
0620 23:29:56.749336 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.749346 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.760585 (+ 11239us) server_negotiation.cc:407] Connection header received
0620 23:29:56.760871 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.760877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.761002 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.761146 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:56.762973 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.764242 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.766342 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.767159 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.767293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.773635 (+ 6342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:56.773662 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:56.773689 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:56.773765 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:56.778343 (+ 4578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.779540 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.779554 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.779558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.779781 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.780234 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.780239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.780243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.780736 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:56.781160 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.781500 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.781771 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:29:56.783033 10953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:56.883384 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10830
I20260620 23:29:56.905457 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10964
Jun 20 23:29:56 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:29:56.925149 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:56.906516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49154 (local address 127.0.96.194:42399)
0620 23:29:56.906773 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:56.906782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:56.906815 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:29:56.906962 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:56.906973 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:56.907066 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:56.907217 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:56.908573 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.909941 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.911397 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:56.912075 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:56.912226 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:56.917825 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:56.917841 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:56.917864 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:56.917907 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:56.921890 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.922796 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.922806 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.922810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.922989 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:56.923381 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:56.923386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:56.923389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:56.923832 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:56.924155 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:56.924480 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:29:56.924711 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:29:56.926183 11087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:57.056512 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10964
I20260620 23:29:57.077913 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 11098
Jun 20 23:29:57 dist-test-slave-v34r krb5kdc[10724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:29:57.095175 11242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:29:57.079085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.0.96.195:37085)
0620 23:29:57.079316 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:29:57.079321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:29:57.079343 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:29:57.079417 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:29:57.079421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:29:57.079490 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:29:57.079607 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:29:57.081191 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:57.081925 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:57.083351 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:29:57.083758 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:29:57.083873 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:29:57.089235 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:29:57.089253 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:29:57.089269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:29:57.089318 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:29:57.092117 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:57.093153 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:57.093161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:57.093163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:57.093312 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:29:57.093707 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:29:57.093711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:29:57.093713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:29:57.094081 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:29:57.094292 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 23:29:57.094703 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 23:29:57.094875 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:29:57.095813 11221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:57.189857 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 11098
I20260620 23:29:57.208958 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 10737
I20260620 23:29:57.210490 10799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:29:57.322422 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 10737
2026-06-20T23:29:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4849 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:29:57 dist-test-slave-v34r krb5kdc[11279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:29:57 dist-test-slave-v34r krb5kdc[11279](info): set up 2 sockets
Jun 20 23:29:57 dist-test-slave-v34r krb5kdc[11279](info): commencing operation
krb5kdc: starting...
W20260620 23:29:58.462067 387 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:29:58 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:29:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:29:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/rangeradmin_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/rangeradmin_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/rangerlookup_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/rangerlookup_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
I20260620 23:29:58.846232 387 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWbSr8K/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/postgres -l logfile start
2026-06-20 23:30:01.528 UTC [11309] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:30:01.528 UTC [11309] LOG: listening on IPv4 address "127.0.96.212", port 36293
2026-06-20 23:30:01.532 UTC [11309] LOG: listening on Unix socket "/tmp/.s.PGSQL.36293"
2026-06-20 23:30:01.558 UTC [11312] LOG: database system was shut down at 2026-06-20 23:29:59 UTC
2026-06-20 23:30:01.562 UTC [11309] LOG: database system is ready to accept connections
I20260620 23:30:02.545620 387 mini_postgres.cc:96] Postgres bound to 36293
2026-06-20 23:30:02.567 UTC [11320] FATAL: database "slave" does not exist
127.0.96.212:36293 - accepting connections
I20260620 23:30:02.567945 387 mini_ranger.cc:161] Starting Ranger...
I20260620 23:30:02.600890 387 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:30:03.110108 387 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:30:03.110280 387 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-admin
2026-06-20 23:30:03,425 [I] DB FLAVOR :POSTGRES
2026-06-20 23:30:03,426 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:30:03,426 [I] Checking connection..
2026-06-20 23:30:03,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:30:03,723 [I] Checking connection passed.
2026-06-20 23:30:03,724 [I] --------- Verifying version history table ---------
2026-06-20 23:30:03,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:30:04,003 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:30:04,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:30:04,276 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:30:04,276 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:30:04,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:30:04.554 UTC [11384] WARNING: there is no transaction in progress
2026-06-20 23:30:04,562 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:30:04,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:30:04,866 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:30:04,867 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:30:04,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:30:05,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:30:05,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v34r.c.gcp-upstream.internal','N') ;"
2026-06-20 23:30:05,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:30:05,962 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:30:05,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:30:06,221 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:30:06,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:30:06,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:30:06,822 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:30:06,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:30:07.975 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:07.993 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.015 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.034 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.055 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.077 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.151 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.159 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.175 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.186 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.199 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.216 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.226 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.234 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.241 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:08.249 UTC [11492] WARNING: there is no transaction in progress
2026-06-20 23:30:09,062 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:30:09,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:30:09,353 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:30:09,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:30:09,667 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:30:09,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:30:09,948 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:30:09,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:30:10,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:30:10,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v34r.c.gcp-upstream.internal';"
2026-06-20 23:30:10,792 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:30:10,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:30:11,064 [I] DB_PATCHES have already been applied
I20260620 23:30:11.071067 387 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWbSr8K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:30:11.071161 387 mini_ranger.cc:193] Using host: 127.0.96.212
I20260620 23:30:11.090356 387 mini_ranger.cc:227] Ranger admin URL: http://127.0.96.212:37575
Jun 20, 2026 11:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:30:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:30:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:30:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:30:12.290506 387 mini_ranger.cc:160] Time spent starting Ranger: real 9.723s user 0.000s sys 0.051s
Jun 20, 2026 11:30:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:30:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 11:30:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:30:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:30:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:30:37.695698 387 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/rangerkms_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/rangerkms_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.96.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.96.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:30:37 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998237, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:30:37.907744 387 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:30:37.937104 387 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:30:38.857208 387 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:30:38.857453 387 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms
I20260620 23:30:38.857632 387 mini_ranger_kms.cc:227] Using postgres at 127.0.96.212:36293
2026-06-20 23:30:39,351 [I] DB FLAVOR :POSTGRES
2026-06-20 23:30:39,352 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:30:39,352 [I] Checking connection
2026-06-20 23:30:39,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:30:39,636 [I] connection success
2026-06-20 23:30:39,637 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:30:39,637 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:30:39,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:30:39,938 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:30:39,939 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:30:39,939 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:30:39,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:36293/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:30:40,235 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:30:40.751607 387 mini_ranger_kms.cc:310] Created kms service
I20260620 23:30:40.974110 387 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:30:41.023721 387 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:30:41 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998237, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:30:41.507916 387 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:30:41.508306 387 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:30:41.508390 387 mini_ranger_kms.cc:240] Using host: 127.0.96.212
I20260620 23:30:41.533159 387 mini_ranger_kms.cc:276] Ranger KMS PID: 11673
I20260620 23:30:41.533453 387 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.96.212:41457
Jun 20, 2026 11:30:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:30:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:30:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:30:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:30:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:30:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:30:43.941581 387 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.034s user 0.118s sys 0.415s
I20260620 23:30:43.941895 387 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:30:43.942134 387 mini_ranger_kms.cc:401] 127.0.96.212:41457/kms/v1/keys
Jun 20 23:30:44 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.96.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:30:44 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.96.212@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
Jun 20 23:30:45 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998244, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.96.212@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
Jun 20 23:30:47 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:30:47.731587 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:41499
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:41499
--ranger_config_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41457/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:30:48.178233 11723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:30:48.178642 11723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:30:48.178830 11723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:30:48.187142 11723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:30:48.187316 11723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:30:48.187409 11723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:30:48.187510 11723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:30:48.187582 11723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:30:48.200384 11723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.96.212:41457/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:41499
--ranger_config_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:41499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:30:48.202375 11723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:30:48.204182 11723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:30:48.219393 11731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:48.219460 11728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:48.219516 11729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:30:48.219921 11723 server_base.cc:1034] running on GCE node
I20260620 23:30:48.221320 11723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:30:48.223266 11723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:30:48.224583 11723 hybrid_clock.cc:648] HybridClock initialized: now 1781998248224490 us; error 75 us; skew 500 ppm
I20260620 23:30:48.225140 11723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:30:48 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:30:48.233184 11723 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:30:48.235357 11723 webserver.cc:460] Webserver started at http://127.0.96.254:33559/ using document root <none> and password file <none>
I20260620 23:30:48.236335 11723 fs_manager.cc:359] Metadata directory not provided
I20260620 23:30:48.236495 11723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:30:48.236891 11723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:30:48 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:30:48.309238 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "56d7ad230eee49e2bc272d1d98cf9351"
format_stamp: "Formatted at 2026-06-20 23:30:48 on dist-test-slave-v34r"
server_key: "82629d1481a0e4768c3f18fb7fc8346d"
server_key_iv: "da36237bd56daa05a9d1f31b82f91bfa"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:48.310338 11723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "56d7ad230eee49e2bc272d1d98cf9351"
format_stamp: "Formatted at 2026-06-20 23:30:48 on dist-test-slave-v34r"
server_key: "82629d1481a0e4768c3f18fb7fc8346d"
server_key_iv: "da36237bd56daa05a9d1f31b82f91bfa"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:48.317374 11723 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260620 23:30:48.340839 11738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:48.342891 11723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:30:48.343117 11723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "56d7ad230eee49e2bc272d1d98cf9351"
format_stamp: "Formatted at 2026-06-20 23:30:48 on dist-test-slave-v34r"
server_key: "82629d1481a0e4768c3f18fb7fc8346d"
server_key_iv: "da36237bd56daa05a9d1f31b82f91bfa"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:48.343416 11723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:30:48.368572 11723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:30:48.374593 11723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:30:48.375007 11723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:30:48.400339 11723 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:41499
I20260620 23:30:48.400394 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:41499 every 8 connection(s)
I20260620 23:30:48.402134 11723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:30:48.406500 11790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:30:48.411300 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 11723
I20260620 23:30:48.411576 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:30:48.412277 387 external_mini_cluster.cc:1428] Setting key a848b73eab8ace5ca61532d155e21e47
I20260620 23:30:48.418238 11790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351: Bootstrap starting.
I20260620 23:30:48.422083 11790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351: Neither blocks nor log segments found. Creating new log.
I20260620 23:30:48.423753 11790 log.cc:824] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351: Log is configured to *not* fsync() on all Append() calls
I20260620 23:30:48.427493 11790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351: No bootstrap required, opened a new log
Jun 20 23:30:48 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:30:48.436946 11790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d7ad230eee49e2bc272d1d98cf9351" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 41499 } }
I20260620 23:30:48.437618 11790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:30:48.437776 11790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d7ad230eee49e2bc272d1d98cf9351, State: Initialized, Role: FOLLOWER
I20260620 23:30:48.438556 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:48.414794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.0.96.254:41499)
0620 23:30:48.415882 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:48.415897 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:48.415951 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:30:48.418068 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:48.418100 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:48.418685 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:48.419118 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:48.420908 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:48.422540 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:48.424775 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:48.425636 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:48.425863 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:48.430150 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:48.430163 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:48.430186 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:48.430237 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:48.433809 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:48.434539 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:48.434546 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:48.434557 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:48.434753 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:48.435159 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:48.435164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:48.435174 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:48.435848 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:48.436156 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:48.436686 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 23:30:48.437183 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":525,"threads_started":1}
I20260620 23:30:48.438656 11790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d7ad230eee49e2bc272d1d98cf9351" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 41499 } }
I20260620 23:30:48.439074 11790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:30:48.439248 11790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:30:48.439459 11790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:30:48.443444 11790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d7ad230eee49e2bc272d1d98cf9351" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 41499 } }
I20260620 23:30:48.444154 11790 leader_election.cc:302] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d7ad230eee49e2bc272d1d98cf9351; no voters:
I20260620 23:30:48.444911 11790 leader_election.cc:288] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:30:48.445109 11795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:30:48.446177 11795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [term 1 LEADER]: Becoming Leader. State: Replica: 56d7ad230eee49e2bc272d1d98cf9351, State: Running, Role: LEADER
I20260620 23:30:48.446818 11795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d7ad230eee49e2bc272d1d98cf9351" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 41499 } }
I20260620 23:30:48.447919 11790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:30:48.450058 11797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d7ad230eee49e2bc272d1d98cf9351. Latest consensus state: current_term: 1 leader_uuid: "56d7ad230eee49e2bc272d1d98cf9351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d7ad230eee49e2bc272d1d98cf9351" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 41499 } } }
I20260620 23:30:48.450449 11797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [sys.catalog]: This master's current role is: LEADER
I20260620 23:30:48.451328 11796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d7ad230eee49e2bc272d1d98cf9351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d7ad230eee49e2bc272d1d98cf9351" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 41499 } } }
I20260620 23:30:48.451630 11796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351 [sys.catalog]: This master's current role is: LEADER
I20260620 23:30:48.453405 11801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:30:48.461205 11801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:30:48.462242 11790 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:30:48.484500 11801 catalog_manager.cc:1347] Generated new cluster ID: d7e0a7c9dd504b10afdc0bd830ac7e58
I20260620 23:30:48.484762 11801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:30:48.519294 11801 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:30:48.521420 11801 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:30:48.529345 11801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351: Generated new TSK 0
I20260620 23:30:48.530690 11801 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:30:49.361284 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:41499
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:30:49 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:30:49.840631 11835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:30:49.841077 11835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:30:49.841231 11835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:30:49.853214 11835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:30:49.853371 11835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:30:49.853654 11835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:30:49.869540 11835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.96.212:41457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:30:49.871472 11835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:30:49.873306 11835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:30:49.892709 11845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:49.893379 11846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:49.901247 11848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:30:49.901980 11835 server_base.cc:1034] running on GCE node
I20260620 23:30:49.902887 11835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:30:49.914078 11835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:30:49.915930 11835 hybrid_clock.cc:648] HybridClock initialized: now 1781998249915828 us; error 128 us; skew 500 ppm
I20260620 23:30:49.916411 11835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:30:49 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:30:49.922039 11835 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:30:49.924837 11835 webserver.cc:460] Webserver started at http://127.0.96.193:38601/ using document root <none> and password file <none>
I20260620 23:30:49.926170 11835 fs_manager.cc:359] Metadata directory not provided
I20260620 23:30:49.926370 11835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:30:49.926846 11835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:30:49 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:30:49.955054 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "774d8a3c6cb840a2a75ec0f96b7f231a"
format_stamp: "Formatted at 2026-06-20 23:30:49 on dist-test-slave-v34r"
server_key: "935f3a7c116bf4876835ba33e72ad107"
server_key_iv: "53fca269455decdc5f3b742c28b24ff2"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:49.959880 11835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "774d8a3c6cb840a2a75ec0f96b7f231a"
format_stamp: "Formatted at 2026-06-20 23:30:49 on dist-test-slave-v34r"
server_key: "935f3a7c116bf4876835ba33e72ad107"
server_key_iv: "53fca269455decdc5f3b742c28b24ff2"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:49.967765 11835 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260620 23:30:49.999047 11855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:50.002655 11835 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.008s
I20260620 23:30:50.003233 11835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "774d8a3c6cb840a2a75ec0f96b7f231a"
format_stamp: "Formatted at 2026-06-20 23:30:49 on dist-test-slave-v34r"
server_key: "935f3a7c116bf4876835ba33e72ad107"
server_key_iv: "53fca269455decdc5f3b742c28b24ff2"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:50.003759 11835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:30:50.073096 11835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:30:50.080160 11835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:30:50.080698 11835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:30:50.093947 11835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:30:50.109455 11835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:30:50.109727 11835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:50.110524 11835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:30:50.110749 11835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:30:50 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:30:50.665174 11835 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:36947
I20260620 23:30:50.665184 11967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:36947 every 8 connection(s)
I20260620 23:30:50.667063 11835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:30:50.676154 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 11835
I20260620 23:30:50.676457 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:30:50.677552 387 external_mini_cluster.cc:1428] Setting key b97510563b41dead421f9019cd00fb2d
Jun 20 23:30:50 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:30:50.712598 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:50.672259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:41499 (local address 127.0.96.193:57327)
0620 23:30:50.673387 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:50.673454 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:50.676979 (+ 3525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:50.686892 (+ 9913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:50.686914 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:50.687887 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:30:50.689130 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:50.689172 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:50.691142 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:50.691152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:50.693581 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:50.693605 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:50.695046 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:50.695056 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:50.695484 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:50.696474 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:30:50.696514 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:30:50.700270 (+ 3756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:30:50.705865 (+ 5595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:30:50.705882 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:30:50.705912 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:30:50.706788 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:30:50.707415 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:30:50.707423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:30:50.707430 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:30:50.707816 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:30:50.708655 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:30:50.708668 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:30:50.709356 (+ 688us) client_negotiation.cc:769] Sending connection context
0620 23:30:50.709742 (+ 386us) client_negotiation.cc:240] Negotiation successful
0620 23:30:50.710520 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":465,"threads_started":1}
I20260620 23:30:50.714906 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:50.671376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:57327 (local address 127.0.96.254:41499)
0620 23:30:50.685687 (+ 14311us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:50.685698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:50.685737 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:30:50.685832 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:50.685842 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:50.685962 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:50.686148 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:50.689503 (+ 3355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:50.690853 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:50.693849 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:50.694662 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:50.696090 (+ 1428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:50.700704 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:50.700715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:50.700724 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:50.700782 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:50.705568 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:50.707037 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:50.707045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:50.707052 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:50.707253 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:50.708007 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:50.708015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:50.708021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:50.708481 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:50.713754 (+ 5273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:50.714014 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 23:30:50.714423 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14077,"thread_start_us":13823,"threads_started":1}
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:30:50.717604 11968 heartbeater.cc:347] Connected to a master server at 127.0.96.254:41499
I20260620 23:30:50.718192 11968 heartbeater.cc:464] Registering TS with master...
I20260620 23:30:50.719659 11968 heartbeater.cc:511] Master 127.0.96.254:41499 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:30:50.722647 11755 ts_manager.cc:194] Registered new tserver with Master: 774d8a3c6cb840a2a75ec0f96b7f231a (127.0.96.193:36947)
I20260620 23:30:50.724787 11755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:57327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:30:50.758700 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:41499
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:30:50 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
W20260620 23:30:51.305601 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:30:51.305989 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:30:51.306169 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:30:51.315338 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:30:51.315495 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:30:51.315698 11975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:30:51.328095 11975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.96.212:41457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:30:51.330046 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:30:51.331739 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:30:51.351564 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:51.351611 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:51.354312 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:30:51.355230 11975 server_base.cc:1034] running on GCE node
I20260620 23:30:51.356114 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:30:51.357517 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:30:51.358875 11975 hybrid_clock.cc:648] HybridClock initialized: now 1781998251358789 us; error 85 us; skew 500 ppm
I20260620 23:30:51.359316 11975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:30:51 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:30:51.363808 11975 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:30:51.366828 11975 webserver.cc:460] Webserver started at http://127.0.96.194:45185/ using document root <none> and password file <none>
I20260620 23:30:51.367892 11975 fs_manager.cc:359] Metadata directory not provided
I20260620 23:30:51.368075 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:30:51.368527 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:30:51 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:30:51.392027 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "65d104d0a9f14a50add9c90d6af18f08"
format_stamp: "Formatted at 2026-06-20 23:30:51 on dist-test-slave-v34r"
server_key: "845e0a5147e3ec0d643c289ff362da80"
server_key_iv: "62b23b1045dae98f28b339f431e97f3e"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:51.393651 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "65d104d0a9f14a50add9c90d6af18f08"
format_stamp: "Formatted at 2026-06-20 23:30:51 on dist-test-slave-v34r"
server_key: "845e0a5147e3ec0d643c289ff362da80"
server_key_iv: "62b23b1045dae98f28b339f431e97f3e"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:51.413496 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.006s sys 0.012s
I20260620 23:30:51.436861 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:51.439026 11975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:30:51.439388 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "65d104d0a9f14a50add9c90d6af18f08"
format_stamp: "Formatted at 2026-06-20 23:30:51 on dist-test-slave-v34r"
server_key: "845e0a5147e3ec0d643c289ff362da80"
server_key_iv: "62b23b1045dae98f28b339f431e97f3e"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:51.439741 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:30:51.477810 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:30:51.482878 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:30:51.483246 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:30:51.484891 11975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:30:51.486636 11975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:30:51.486759 11975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:51.486924 11975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:30:51.486999 11975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:51.531596 11975 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:36065
I20260620 23:30:51.531642 12110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:36065 every 8 connection(s)
I20260620 23:30:51.533324 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:30:51.534557 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 11975
I20260620 23:30:51.534768 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:30:51.535266 387 external_mini_cluster.cc:1428] Setting key ae74207b6dc9c6274e1602b5d948f0aa
Jun 20 23:30:51 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:30:51.563906 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:51.536626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:42813 (local address 127.0.96.254:41499)
0620 23:30:51.537368 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:51.537377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:51.539158 (+ 1781us) server_negotiation.cc:407] Connection header received
0620 23:30:51.542542 (+ 3384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:51.542549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:51.542674 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:51.542813 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:51.544918 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:51.546367 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:51.548827 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:51.549740 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:51.549956 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:51.554575 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:51.554612 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:51.554622 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:51.554695 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:51.559247 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:51.560342 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:51.560352 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:51.560358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:51.560603 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:51.561319 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:51.561326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:51.561330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:51.561837 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:51.562020 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:51.562988 (+ 968us) server_negotiation.cc:299] Negotiation successful
0620 23:30:51.563320 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":382,"threads_started":1}
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:30:51.564590 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:51.537465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:41499 (local address 127.0.96.194:42813)
0620 23:30:51.538892 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:51.538957 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:51.542123 (+ 3166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:51.543141 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:51.543152 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:51.543733 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:30:51.544645 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:51.544707 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:51.546631 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:51.546638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:51.548323 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:51.548353 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:51.550039 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:51.550046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:51.550323 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:51.551273 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:30:51.551298 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:30:51.554136 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:30:51.559548 (+ 5412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:30:51.559557 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:30:51.559575 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:30:51.560144 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:30:51.560809 (+ 665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:30:51.560813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:30:51.560816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:30:51.561174 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:30:51.562047 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:30:51.562055 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:30:51.562488 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 23:30:51.562760 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 23:30:51.563217 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"thread_start_us":529,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:30:51.567022 12111 heartbeater.cc:347] Connected to a master server at 127.0.96.254:41499
I20260620 23:30:51.567433 12111 heartbeater.cc:464] Registering TS with master...
I20260620 23:30:51.568436 12111 heartbeater.cc:511] Master 127.0.96.254:41499 requested a full tablet report, sending...
I20260620 23:30:51.577576 11755 ts_manager.cc:194] Registered new tserver with Master: 65d104d0a9f14a50add9c90d6af18f08 (127.0.96.194:36065)
I20260620 23:30:51.578652 11755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:42813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:30:51.605620 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41457/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:41499
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:36443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:30:51.729004 11968 heartbeater.cc:503] Master 127.0.96.254:41499 was elected leader, sending a full tablet report...
W20260620 23:30:52.027877 12117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:30:52.028227 12117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:30:52.028384 12117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:30:52.036394 12117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:30:52.036566 12117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:30:52.036755 12117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:30:52.047333 12117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:36443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.96.212:41457/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:41499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:30:52.048867 12117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:30:52.050422 12117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:30:52.066224 12122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:52.066424 12125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:30:52.066238 12123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:30:52.068239 12117 server_base.cc:1034] running on GCE node
I20260620 23:30:52.068917 12117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:30:52.069983 12117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:30:52.071197 12117 hybrid_clock.cc:648] HybridClock initialized: now 1781998252071145 us; error 49 us; skew 500 ppm
I20260620 23:30:52.071557 12117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:30:52 dist-test-slave-v34r krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:30:52.075126 12117 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:30:52.076758 12117 webserver.cc:460] Webserver started at http://127.0.96.195:42377/ using document root <none> and password file <none>
I20260620 23:30:52.077548 12117 fs_manager.cc:359] Metadata directory not provided
I20260620 23:30:52.077654 12117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:30:52.077924 12117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:30:52 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:30:52.096529 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "d4ce62f12e0d4f75933eac781e53269d"
format_stamp: "Formatted at 2026-06-20 23:30:52 on dist-test-slave-v34r"
server_key: "657500bc7b61790f954e3713220b6168"
server_key_iv: "9043ea2f73a2ad84254666bd23e7ab12"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:52.097534 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "d4ce62f12e0d4f75933eac781e53269d"
format_stamp: "Formatted at 2026-06-20 23:30:52 on dist-test-slave-v34r"
server_key: "657500bc7b61790f954e3713220b6168"
server_key_iv: "9043ea2f73a2ad84254666bd23e7ab12"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:52.104138 12117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:30:52.124639 12132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:52.126358 12117 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:30:52.126538 12117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "d4ce62f12e0d4f75933eac781e53269d"
format_stamp: "Formatted at 2026-06-20 23:30:52 on dist-test-slave-v34r"
server_key: "657500bc7b61790f954e3713220b6168"
server_key_iv: "9043ea2f73a2ad84254666bd23e7ab12"
server_key_version: "kuduclusterkey@0"
I20260620 23:30:52.126776 12117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:30:52.168000 12117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:30:52.172326 12117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:30:52.172760 12117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:30:52.173970 12117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:30:52.175535 12117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:30:52.175647 12117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:52.175812 12117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:30:52.175872 12117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:30:52.220927 12117 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:37611
I20260620 23:30:52.221072 12244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:37611 every 8 connection(s)
I20260620 23:30:52.222416 12117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:30:52.227351 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 12117
I20260620 23:30:52.227725 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:30:52.228245 387 external_mini_cluster.cc:1428] Setting key 4f5f2a96514b5325bf641d3908214b42
Jun 20 23:30:52 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:30:52.250036 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.226033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:54141 (local address 127.0.96.254:41499)
0620 23:30:52.226741 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.226747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.227769 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 23:30:52.229879 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.229887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.230057 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.230221 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:52.232912 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.233804 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.236054 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.236540 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.236658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.241228 (+ 4570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:52.241264 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:52.241268 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:52.241315 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:52.245625 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.246894 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.246899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.246901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.247058 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.247878 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.247882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.247885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.248248 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:52.248380 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.249470 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.249702 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":267,"threads_started":1}
I20260620 23:30:52.251389 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.226636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:41499 (local address 127.0.96.195:54141)
0620 23:30:52.227559 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:52.227614 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:52.229488 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:52.230839 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:52.230857 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:52.231647 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:30:52.232658 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.232685 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.234022 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.234028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.235798 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.235855 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.236851 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.236857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.237393 (+ 536us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.238329 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:30:52.238360 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:30:52.240836 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:30:52.245998 (+ 5162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:30:52.246014 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:30:52.246042 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:30:52.246742 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:30:52.247317 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:30:52.247321 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:30:52.247324 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:30:52.247725 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:30:52.248428 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:30:52.248436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:30:52.249051 (+ 615us) client_negotiation.cc:769] Sending connection context
0620 23:30:52.249371 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 23:30:52.249969 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":414,"threads_started":1}
I20260620 23:30:52.253432 12245 heartbeater.cc:347] Connected to a master server at 127.0.96.254:41499
I20260620 23:30:52.253932 12245 heartbeater.cc:464] Registering TS with master...
I20260620 23:30:52.255017 12245 heartbeater.cc:511] Master 127.0.96.254:41499 requested a full tablet report, sending...
I20260620 23:30:52.257520 11755 ts_manager.cc:194] Registered new tserver with Master: d4ce62f12e0d4f75933eac781e53269d (127.0.96.195:37611)
I20260620 23:30:52.258678 11755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:54141
I20260620 23:30:52.259478 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:30:52.280031 12247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.265833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.0.96.254:41499)
0620 23:30:52.266108 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.266115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.266402 (+ 287us) server_negotiation.cc:407] Connection header received
0620 23:30:52.266580 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.266585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.266670 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.266772 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:52.268351 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.269537 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.271071 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.271493 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.271639 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.274441 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:52.274448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:52.274455 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:52.274503 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:52.277494 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.278297 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.278300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.278303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.278418 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.278796 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.278800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.278802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.279087 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:52.279320 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.279604 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.279802 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 23:30:52.285159 11755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:30:52.292579 11755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:30:52.321580 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.309275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37611 (local address 127.0.0.1:43022)
0620 23:30:52.311390 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:52.311428 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:52.311693 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:52.312909 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:52.312918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:52.312950 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:30:52.313397 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.313408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.316133 (+ 2725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.316138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.318504 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.318524 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.320457 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.320468 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.320712 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.320787 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 23:30:52.320927 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:30:52.321174 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1882,"thread_start_us":2804,"threads_started":1}
I20260620 23:30:52.322918 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.309557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.0.96.195:37611)
0620 23:30:52.310263 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.310276 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.311571 (+ 1295us) server_negotiation.cc:407] Connection header received
0620 23:30:52.311976 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.311986 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.312327 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.312519 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:30:52.313650 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.315931 (+ 2281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.318685 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.320188 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.320338 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.322214 (+ 1876us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.322300 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.322523 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":898,"threads_started":1}
I20260620 23:30:52.323042 12258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.309962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:36065 (local address 127.0.0.1:36086)
0620 23:30:52.312442 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:52.312464 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:52.312614 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:52.313530 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:52.313544 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:52.313729 (+ 185us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:30:52.315083 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.315099 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.317619 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.317629 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.320196 (+ 2567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.320218 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.322314 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.322322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.322456 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.322477 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 23:30:52.322586 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 23:30:52.322711 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2316,"thread_start_us":1037,"threads_started":1}
I20260620 23:30:52.323132 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.310222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.0.96.194:36065)
0620 23:30:52.310830 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.310840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.312799 (+ 1959us) server_negotiation.cc:407] Connection header received
0620 23:30:52.312921 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.312929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.313295 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.313516 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:30:52.315281 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.317374 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.320473 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.321948 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.322052 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.322293 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.322627 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.322790 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":451,"threads_started":1}
I20260620 23:30:52.327651 12180 tablet_service.cc:1467] Processing CreateTablet for tablet a55e714a950c4ee7bdbaeff982c4b6b6 (DEFAULT_TABLE table=test-table [id=91f96b3ced814258a6717140556accf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:30:52.327648 12046 tablet_service.cc:1467] Processing CreateTablet for tablet a55e714a950c4ee7bdbaeff982c4b6b6 (DEFAULT_TABLE table=test-table [id=91f96b3ced814258a6717140556accf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:30:52.329197 12180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55e714a950c4ee7bdbaeff982c4b6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:30:52.329198 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55e714a950c4ee7bdbaeff982c4b6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:30:52.330148 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.313516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36947 (local address 127.0.0.1:35022)
0620 23:30:52.319253 (+ 5737us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:52.319274 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:52.319418 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:52.320115 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:52.320126 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:52.320153 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:30:52.320675 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.320687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.324665 (+ 3978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.324671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.327332 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.327348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.329106 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.329114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.329411 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.329439 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:30:52.329577 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 23:30:52.329741 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5566,"thread_start_us":5384,"threads_started":1}
I20260620 23:30:52.330205 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.312899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.0.96.193:36947)
0620 23:30:52.314251 (+ 1352us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.314264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.319405 (+ 5141us) server_negotiation.cc:407] Connection header received
0620 23:30:52.319601 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.319609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.319878 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.320090 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:30:52.320956 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.322998 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.327555 (+ 4557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.328830 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.328995 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.329316 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.329647 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.329837 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":1064,"threads_started":1}
I20260620 23:30:52.335127 11903 tablet_service.cc:1467] Processing CreateTablet for tablet a55e714a950c4ee7bdbaeff982c4b6b6 (DEFAULT_TABLE table=test-table [id=91f96b3ced814258a6717140556accf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:30:52.336767 11903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a55e714a950c4ee7bdbaeff982c4b6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:30:52.345641 12264 tablet_bootstrap.cc:492] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Bootstrap starting.
I20260620 23:30:52.346112 12265 tablet_bootstrap.cc:492] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Bootstrap starting.
I20260620 23:30:52.350441 12264 tablet_bootstrap.cc:654] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Neither blocks nor log segments found. Creating new log.
I20260620 23:30:52.350664 12265 tablet_bootstrap.cc:654] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Neither blocks nor log segments found. Creating new log.
I20260620 23:30:52.351843 12264 log.cc:824] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:30:52.352227 12265 log.cc:824] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Log is configured to *not* fsync() on all Append() calls
I20260620 23:30:52.354426 12268 tablet_bootstrap.cc:492] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Bootstrap starting.
I20260620 23:30:52.356154 12265 tablet_bootstrap.cc:492] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: No bootstrap required, opened a new log
I20260620 23:30:52.356441 12265 ts_tablet_manager.cc:1397] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:30:52.358572 12268 tablet_bootstrap.cc:654] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Neither blocks nor log segments found. Creating new log.
I20260620 23:30:52.359704 12264 tablet_bootstrap.cc:492] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: No bootstrap required, opened a new log
I20260620 23:30:52.359869 12268 log.cc:824] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:30:52.360004 12264 ts_tablet_manager.cc:1397] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20260620 23:30:52.364267 12268 tablet_bootstrap.cc:492] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: No bootstrap required, opened a new log
I20260620 23:30:52.364560 12268 ts_tablet_manager.cc:1397] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:30:52.365911 12265 raft_consensus.cc:348] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.366395 12265 raft_consensus.cc:374] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:30:52.366515 12265 raft_consensus.cc:729] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d104d0a9f14a50add9c90d6af18f08, State: Initialized, Role: FOLLOWER
I20260620 23:30:52.367262 12265 consensus_queue.cc:260] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.368602 12111 heartbeater.cc:503] Master 127.0.96.254:41499 was elected leader, sending a full tablet report...
I20260620 23:30:52.369451 12265 ts_tablet_manager.cc:1428] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260620 23:30:52.371281 12264 raft_consensus.cc:348] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.371905 12264 raft_consensus.cc:374] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:30:52.372042 12264 raft_consensus.cc:729] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ce62f12e0d4f75933eac781e53269d, State: Initialized, Role: FOLLOWER
I20260620 23:30:52.372772 12264 consensus_queue.cc:260] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.374655 12245 heartbeater.cc:503] Master 127.0.96.254:41499 was elected leader, sending a full tablet report...
I20260620 23:30:52.375262 12264 ts_tablet_manager.cc:1428] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260620 23:30:52.375598 12268 raft_consensus.cc:348] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.375954 12268 raft_consensus.cc:374] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:30:52.376048 12268 raft_consensus.cc:729] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774d8a3c6cb840a2a75ec0f96b7f231a, State: Initialized, Role: FOLLOWER
I20260620 23:30:52.376719 12268 consensus_queue.cc:260] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.378728 12268 ts_tablet_manager.cc:1428] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260620 23:30:52.424594 11969 tablet.cc:2369] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:30:52.474334 12246 tablet.cc:2369] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:30:52.518847 12270 raft_consensus.cc:482] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:30:52.519207 12270 raft_consensus.cc:504] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.521175 12270 leader_election.cc:288] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4ce62f12e0d4f75933eac781e53269d (127.0.96.195:37611), 774d8a3c6cb840a2a75ec0f96b7f231a (127.0.96.193:36947)
I20260620 23:30:52.532696 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.521846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:37829 (local address 127.0.96.195:37611)
0620 23:30:52.522797 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.522807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.523231 (+ 424us) server_negotiation.cc:407] Connection header received
0620 23:30:52.523537 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.523545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.523659 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.523737 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:30:52.524761 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.526056 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.529848 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.531403 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.531576 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.531659 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.532205 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.532407 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743}
I20260620 23:30:52.533845 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.521504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:37611 (local address 127.0.96.194:37829)
0620 23:30:52.523070 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:52.523094 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:52.523332 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:52.523982 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:52.523987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:52.524018 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:30:52.524564 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.524578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.526232 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.526238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.529547 (+ 3309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.529571 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.531683 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.531691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.531925 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.531976 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:30:52.533051 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0620 23:30:52.533266 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"mutex_wait_us":5,"thread_start_us":1136,"threads_started":1}
I20260620 23:30:52.534178 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.524320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:46133 (local address 127.0.96.193:36947)
0620 23:30:52.524570 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.524575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.524815 (+ 240us) server_negotiation.cc:407] Connection header received
0620 23:30:52.525197 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.525203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.525287 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.525395 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:30:52.526191 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.527505 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.531162 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.532537 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.532697 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.532779 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.533774 (+ 995us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.533905 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:30:52.534646 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.523120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36947 (local address 127.0.96.194:46133)
0620 23:30:52.524663 (+ 1543us) negotiation.cc:107] Waiting for socket to connect
0620 23:30:52.524688 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:30:52.524961 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:30:52.525444 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:30:52.525448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:30:52.525471 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:30:52.525940 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.525956 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.527724 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.527729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.530959 (+ 3230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:30:52.530980 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.533298 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.533309 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:30:52.533561 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.533605 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:30:52.533744 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 23:30:52.533903 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"thread_start_us":1494,"threads_started":1}
I20260620 23:30:52.535209 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55e714a950c4ee7bdbaeff982c4b6b6" candidate_uuid: "65d104d0a9f14a50add9c90d6af18f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ce62f12e0d4f75933eac781e53269d" is_pre_election: true
I20260620 23:30:52.535808 12200 raft_consensus.cc:2454] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d104d0a9f14a50add9c90d6af18f08 in term 0.
W20260620 23:30:52.536001 12112 tablet.cc:2369] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:30:52.535883 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55e714a950c4ee7bdbaeff982c4b6b6" candidate_uuid: "65d104d0a9f14a50add9c90d6af18f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" is_pre_election: true
I20260620 23:30:52.536386 11923 raft_consensus.cc:2454] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d104d0a9f14a50add9c90d6af18f08 in term 0.
I20260620 23:30:52.536700 12001 leader_election.cc:302] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65d104d0a9f14a50add9c90d6af18f08, d4ce62f12e0d4f75933eac781e53269d; no voters:
I20260620 23:30:52.537276 12270 raft_consensus.cc:2781] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:30:52.537434 12270 raft_consensus.cc:482] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:30:52.537557 12270 raft_consensus.cc:3037] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:30:52.540468 12270 raft_consensus.cc:504] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.541185 12270 leader_election.cc:288] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [CANDIDATE]: Term 1 election: Requested vote from peers d4ce62f12e0d4f75933eac781e53269d (127.0.96.195:37611), 774d8a3c6cb840a2a75ec0f96b7f231a (127.0.96.193:36947)
I20260620 23:30:52.541760 11923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55e714a950c4ee7bdbaeff982c4b6b6" candidate_uuid: "65d104d0a9f14a50add9c90d6af18f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a"
I20260620 23:30:52.541971 11923 raft_consensus.cc:3037] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:30:52.541904 12200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a55e714a950c4ee7bdbaeff982c4b6b6" candidate_uuid: "65d104d0a9f14a50add9c90d6af18f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4ce62f12e0d4f75933eac781e53269d"
I20260620 23:30:52.542209 12200 raft_consensus.cc:3037] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:30:52.544298 11923 raft_consensus.cc:2454] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d104d0a9f14a50add9c90d6af18f08 in term 1.
I20260620 23:30:52.544783 12001 leader_election.cc:302] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65d104d0a9f14a50add9c90d6af18f08, 774d8a3c6cb840a2a75ec0f96b7f231a; no voters:
I20260620 23:30:52.545128 12270 raft_consensus.cc:2781] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:30:52.545657 12270 raft_consensus.cc:686] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 1 LEADER]: Becoming Leader. State: Replica: 65d104d0a9f14a50add9c90d6af18f08, State: Running, Role: LEADER
I20260620 23:30:52.546304 12200 raft_consensus.cc:2454] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d104d0a9f14a50add9c90d6af18f08 in term 1.
I20260620 23:30:52.546294 12270 consensus_queue.cc:237] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } }
I20260620 23:30:52.552196 11754 catalog_manager.cc:5495] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65d104d0a9f14a50add9c90d6af18f08 (127.0.96.194). New cstate: current_term: 1 leader_uuid: "65d104d0a9f14a50add9c90d6af18f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65d104d0a9f14a50add9c90d6af18f08" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 36065 } health_report { overall_health: HEALTHY } } }
I20260620 23:30:52.626590 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.620296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.0.96.194:36065)
0620 23:30:52.620507 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.620511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.620593 (+ 82us) server_negotiation.cc:407] Connection header received
0620 23:30:52.620847 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.620851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.620935 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.621174 (+ 239us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:30:52.621605 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.622423 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.624276 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.624720 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.624851 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.625478 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:30:52.626024 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:30:52.626147 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.626309 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.626378 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:30:52.637140 12200 raft_consensus.cc:1261] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 1 FOLLOWER]: Refusing update from remote peer 65d104d0a9f14a50add9c90d6af18f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:30:52.637140 11923 raft_consensus.cc:1261] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 1 FOLLOWER]: Refusing update from remote peer 65d104d0a9f14a50add9c90d6af18f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:30:52.638366 12275 consensus_queue.cc:1035] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4ce62f12e0d4f75933eac781e53269d" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 37611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:30:52.638842 12270 consensus_queue.cc:1035] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "774d8a3c6cb840a2a75ec0f96b7f231a" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:30:52.678745 11755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:30:52.679272 11755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:30:52.686949 11755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d7ad230eee49e2bc272d1d98cf9351: Sending DeleteTablet for 3 replicas of tablet a55e714a950c4ee7bdbaeff982c4b6b6
I20260620 23:30:52.688736 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet a55e714a950c4ee7bdbaeff982c4b6b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:30:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:36086
I20260620 23:30:52.688898 11903 tablet_service.cc:1514] Processing DeleteTablet for tablet a55e714a950c4ee7bdbaeff982c4b6b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:30:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:35022
I20260620 23:30:52.689514 12180 tablet_service.cc:1514] Processing DeleteTablet for tablet a55e714a950c4ee7bdbaeff982c4b6b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:30:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:43022
I20260620 23:30:52.690364 12289 tablet_replica.cc:330] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: stopping tablet replica
I20260620 23:30:52.690963 12289 raft_consensus.cc:2229] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:30:52.691413 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 11835
I20260620 23:30:52.691574 12289 raft_consensus.cc:2258] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:30:52.698377 12290 tablet_replica.cc:330] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: stopping tablet replica
I20260620 23:30:52.699201 12288 tablet_replica.cc:330] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: stopping tablet replica
I20260620 23:30:52.699884 12290 raft_consensus.cc:2229] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:30:52.700475 12288 raft_consensus.cc:2229] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:30:52.700608 12289 ts_tablet_manager.cc:1905] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:30:52.701246 12290 raft_consensus.cc:2258] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:30:52.702081 12288 raft_consensus.cc:2258] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:30:52.704834 12288 ts_tablet_manager.cc:1905] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:30:52.704831 12290 ts_tablet_manager.cc:1905] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:30:52.710007 12289 ts_tablet_manager.cc:1918] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:30:52.710282 12289 log.cc:1196] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/a55e714a950c4ee7bdbaeff982c4b6b6
I20260620 23:30:52.710952 12289 ts_tablet_manager.cc:1939] T a55e714a950c4ee7bdbaeff982c4b6b6 P 774d8a3c6cb840a2a75ec0f96b7f231a: Deleting consensus metadata
I20260620 23:30:52.713220 11741 catalog_manager.cc:4841] TS 774d8a3c6cb840a2a75ec0f96b7f231a (127.0.96.193:36947): tablet a55e714a950c4ee7bdbaeff982c4b6b6 (table test-table [id=91f96b3ced814258a6717140556accf1]) successfully deleted
I20260620 23:30:52.714118 12290 ts_tablet_manager.cc:1918] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:30:52.714403 12290 log.cc:1196] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/a55e714a950c4ee7bdbaeff982c4b6b6
I20260620 23:30:52.715026 12290 ts_tablet_manager.cc:1939] T a55e714a950c4ee7bdbaeff982c4b6b6 P d4ce62f12e0d4f75933eac781e53269d: Deleting consensus metadata
Jun 20 23:30:52 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:30:52.718667 11741 catalog_manager.cc:4841] TS d4ce62f12e0d4f75933eac781e53269d (127.0.96.195:37611): tablet a55e714a950c4ee7bdbaeff982c4b6b6 (table test-table [id=91f96b3ced814258a6717140556accf1]) successfully deleted
I20260620 23:30:52.720212 12288 ts_tablet_manager.cc:1918] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:30:52.720391 12288 log.cc:1196] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/a55e714a950c4ee7bdbaeff982c4b6b6
I20260620 23:30:52.721473 12288 ts_tablet_manager.cc:1939] T a55e714a950c4ee7bdbaeff982c4b6b6 P 65d104d0a9f14a50add9c90d6af18f08: Deleting consensus metadata
I20260620 23:30:52.723436 11740 catalog_manager.cc:4841] TS 65d104d0a9f14a50add9c90d6af18f08 (127.0.96.194:36065): tablet a55e714a950c4ee7bdbaeff982c4b6b6 (table test-table [id=91f96b3ced814258a6717140556accf1]) successfully deleted
I20260620 23:30:52.726866 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.692937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.96.193:36947)
0620 23:30:52.693181 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.693191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.697566 (+ 4375us) server_negotiation.cc:407] Connection header received
0620 23:30:52.697803 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.697812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.697919 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.698038 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:52.700221 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.701463 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.706260 (+ 4797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.707741 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.707877 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.719416 (+ 11539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:52.719437 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:52.719457 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:52.719508 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:52.723157 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.724243 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.724257 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.724267 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.724416 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.724861 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.724866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.724870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.725452 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:52.725810 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.726185 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.726445 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:30:52.727753 11963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:30:52.829727 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 11835
I20260620 23:30:52.856434 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 11975
Jun 20 23:30:52 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:30:52.877053 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:52.857601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36096 (local address 127.0.96.194:36065)
0620 23:30:52.857873 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:52.857880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:52.857899 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:30:52.857971 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:52.857975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:52.858044 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:52.858166 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:52.859719 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.860768 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.863030 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:52.863584 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:52.863714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:52.869499 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:52.869519 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:52.869546 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:52.869606 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:52.873548 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.874472 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.874481 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.874484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.874715 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:52.875153 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:52.875158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:52.875162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:52.875617 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:52.876012 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:52.876306 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:30:52.876573 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:30:52.877931 12106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:30:52.999361 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 11975
I20260620 23:30:53.018173 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 12117
Jun 20 23:30:53 dist-test-slave-v34r krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:30:53.036341 12259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:30:53.019151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.96.195:37611)
0620 23:30:53.019350 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:30:53.019355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:30:53.019377 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:30:53.019451 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:30:53.019456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:30:53.019531 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:30:53.019656 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:30:53.021157 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:53.022009 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:53.023453 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:30:53.023939 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:30:53.024072 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:30:53.029358 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:30:53.029378 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:30:53.029401 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:30:53.029461 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:30:53.033162 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:53.034069 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:53.034079 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:53.034082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:53.034289 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:30:53.034737 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:30:53.034741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:30:53.034743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:30:53.035204 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:30:53.035476 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:30:53.035768 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 23:30:53.035985 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:30:53.037326 12240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:30:53.138939 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 12117
I20260620 23:30:53.158238 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 11723
I20260620 23:30:53.159657 11785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:30:53.275727 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 11723
I20260620 23:30:53.305513 387 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:30:53.642583 387 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:30:53.642856 387 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:30:53.989203 387 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:30:53.989 UTC [11309] LOG: received smart shutdown request
2026-06-20 23:30:53.993 UTC [11309] LOG: background worker "logical replication launcher" (PID 11318) exited with exit code 1
2026-06-20 23:30:53.993 UTC [11313] LOG: shutting down
2026-06-20 23:30:54.143 UTC [11309] LOG: database system is shut down
2026-06-20T23:30:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (56950 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:30:54 dist-test-slave-v34r krb5kdc[12304](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:30:54 dist-test-slave-v34r krb5kdc[12304](info): set up 2 sockets
Jun 20 23:30:54 dist-test-slave-v34r krb5kdc[12304](info): commencing operation
krb5kdc: starting...
W20260620 23:30:55.413630 387 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:30:55 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:30:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:30:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/rangeradmin_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/rangeradmin_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/rangerlookup_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/rangerlookup_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
I20260620 23:30:55.781277 387 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWbSr8K/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/postgres -l logfile start
2026-06-20 23:30:58.392 UTC [12334] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 23:30:58.392 UTC [12334] LOG: listening on IPv4 address "127.0.96.212", port 50041
2026-06-20 23:30:58.397 UTC [12334] LOG: listening on Unix socket "/tmp/.s.PGSQL.50041"
2026-06-20 23:30:58.420 UTC [12337] LOG: database system was shut down at 2026-06-20 23:30:56 UTC
2026-06-20 23:30:58.423 UTC [12334] LOG: database system is ready to accept connections
I20260620 23:30:59.413061 387 mini_postgres.cc:96] Postgres bound to 50041
2026-06-20 23:30:59.433 UTC [12345] FATAL: database "slave" does not exist
127.0.96.212:50041 - accepting connections
I20260620 23:30:59.434548 387 mini_ranger.cc:161] Starting Ranger...
I20260620 23:30:59.465773 387 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 23:30:59.968415 387 mini_ranger.cc:87] Created ranger Postgres database
I20260620 23:30:59.968540 387 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-admin
2026-06-20 23:31:00,281 [I] DB FLAVOR :POSTGRES
2026-06-20 23:31:00,282 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:31:00,282 [I] Checking connection..
2026-06-20 23:31:00,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 23:31:00,558 [I] Checking connection passed.
2026-06-20 23:31:00,559 [I] --------- Verifying version history table ---------
2026-06-20 23:31:00,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:31:00,814 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:31:00,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:31:01,067 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 23:31:01,068 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 23:31:01,068 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 23:31:01.353 UTC [12409] WARNING: there is no transaction in progress
2026-06-20 23:31:01,361 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 23:31:01,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 23:31:01,637 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 23:31:01,637 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:31:01,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 23:31:01,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 23:31:02,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v34r.c.gcp-upstream.internal','N') ;"
2026-06-20 23:31:02,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:31:02,764 [I] Table x_portal_user does not exist in database ranger
2026-06-20 23:31:02,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:31:03,043 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 23:31:03,043 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:31:03,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:31:03,557 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 23:31:03,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 23:31:04.640 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.657 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.674 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.689 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.707 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.727 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.788 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.795 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.811 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.822 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.833 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.849 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.860 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.867 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.874 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:04.882 UTC [12517] WARNING: there is no transaction in progress
2026-06-20 23:31:05,605 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 23:31:05,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 23:31:05,851 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 23:31:05,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 23:31:06,126 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 23:31:06,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 23:31:06,421 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 23:31:06,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 23:31:06,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 23:31:06,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v34r.c.gcp-upstream.internal';"
2026-06-20 23:31:07,200 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 23:31:07,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 23:31:07,472 [I] DB_PATCHES have already been applied
I20260620 23:31:07.479331 387 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWbSr8K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 23:31:07.479470 387 mini_ranger.cc:193] Using host: 127.0.96.212
I20260620 23:31:07.497303 387 mini_ranger.cc:227] Ranger admin URL: http://127.0.96.212:35221
Jun 20, 2026 11:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:31:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 23:31:08.706220 387 mini_ranger.cc:160] Time spent starting Ranger: real 9.272s user 0.009s sys 0.041s
Jun 20, 2026 11:31:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:31:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:31:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:31:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:31:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 23:31:34.113533 387 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.96.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.96.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/rangerkms_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.96.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/rangerkms_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.96.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.96.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.96.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/HTTP_127.0.96.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 23:31:34 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 23:31:34.343044 387 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 23:31:34.376051 387 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 23:31:35.316187 387 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 23:31:35.316316 387 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms
I20260620 23:31:35.316370 387 mini_ranger_kms.cc:227] Using postgres at 127.0.96.212:50041
2026-06-20 23:31:35,692 [I] DB FLAVOR :POSTGRES
2026-06-20 23:31:35,693 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 23:31:35,693 [I] Checking connection
2026-06-20 23:31:35,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 23:31:35,971 [I] connection success
2026-06-20 23:31:35,971 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 23:31:35,971 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 23:31:35,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 23:31:36,268 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 23:31:36,269 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 23:31:36,269 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 23:31:36,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.96.212:50041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 23:31:36,610 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 23:31:37.230100 387 mini_ranger_kms.cc:310] Created kms service
I20260620 23:31:37.501407 387 mini_ranger_kms.cc:326] Created kudu user
I20260620 23:31:37.556818 387 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 23:31:37 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998294, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:38.065016 387 mini_ranger_kms.cc:384] Added ranger policy
I20260620 23:31:38.065428 387 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWbSr8K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWbSr8K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 23:31:38.065492 387 mini_ranger_kms.cc:240] Using host: 127.0.96.212
I20260620 23:31:38.081729 387 mini_ranger_kms.cc:276] Ranger KMS PID: 12698
I20260620 23:31:38.081911 387 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.96.212:41959
Jun 20, 2026 11:31:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:31:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:31:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:31:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:31:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:31:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 23:31:40.411823 387 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.069s user 0.086s sys 0.316s
I20260620 23:31:40.412145 387 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 23:31:40.412346 387 mini_ranger_kms.cc:401] 127.0.96.212:41959/kms/v1/keys
Jun 20 23:31:40 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998300, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.96.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 23:31:41 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998300, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.96.212@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
Jun 20 23:31:41 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998300, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.96.212@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
Jun 20 23:31:44 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:31:44.265348 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34567
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34567
--ranger_config_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:31:44.687430 12748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:44.687855 12748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:44.687960 12748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:44.696372 12748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:31:44.696542 12748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:44.696601 12748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:44.696667 12748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:31:44.696722 12748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:31:44.708539 12748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.96.212:41959/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34567
--ranger_config_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:44.710388 12748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:44.712118 12748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:44.727463 12753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:44.727551 12754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:44.727553 12756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:44.728518 12748 server_base.cc:1034] running on GCE node
I20260620 23:31:44.729540 12748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:44.731289 12748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:44.732543 12748 hybrid_clock.cc:648] HybridClock initialized: now 1781998304732499 us; error 49 us; skew 500 ppm
I20260620 23:31:44.732939 12748 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:44 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:44.736366 12748 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:31:44.738127 12748 webserver.cc:460] Webserver started at http://127.0.96.254:34969/ using document root <none> and password file <none>
I20260620 23:31:44.738818 12748 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:44.738914 12748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:44.739178 12748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:31:44 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:44.799353 12748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "da6e87fd944847c3805dc44d8b8b3a40"
format_stamp: "Formatted at 2026-06-20 23:31:44 on dist-test-slave-v34r"
server_key: "ed500d4dfcaef250267723ca571688f2"
server_key_iv: "913d332bc49bac96e9b1969e6ba366dd"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:44.800474 12748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "da6e87fd944847c3805dc44d8b8b3a40"
format_stamp: "Formatted at 2026-06-20 23:31:44 on dist-test-slave-v34r"
server_key: "ed500d4dfcaef250267723ca571688f2"
server_key_iv: "913d332bc49bac96e9b1969e6ba366dd"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:44.807673 12748 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 23:31:44.830436 12763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:44.832809 12748 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:31:44.833096 12748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "da6e87fd944847c3805dc44d8b8b3a40"
format_stamp: "Formatted at 2026-06-20 23:31:44 on dist-test-slave-v34r"
server_key: "ed500d4dfcaef250267723ca571688f2"
server_key_iv: "913d332bc49bac96e9b1969e6ba366dd"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:44.833415 12748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:44.867580 12748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:44.872164 12748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:44.872527 12748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:44.894618 12748 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34567
I20260620 23:31:44.894639 12814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34567 every 8 connection(s)
I20260620 23:31:44.896097 12748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:31:44.901593 12815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:44.906450 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 12748
I20260620 23:31:44.906688 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:31:44.907403 387 external_mini_cluster.cc:1428] Setting key c77a2767d684d87a0c5d09e07d3ca2d8
I20260620 23:31:44.916090 12815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Bootstrap starting.
I20260620 23:31:44.920709 12815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:44.922817 12815 log.cc:824] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:31:44 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:44.927641 12815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: No bootstrap required, opened a new log
I20260620 23:31:44.938963 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:44.909929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.0.96.254:34567)
0620 23:31:44.910945 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:44.910960 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:44.911017 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:31:44.914080 (+ 3063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:44.914110 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:44.914649 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:44.915266 (+ 617us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:44.916766 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:44.918236 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:44.919614 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:44.920457 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:44.920683 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:44.926816 (+ 6133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:44.926831 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:44.926857 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:44.926931 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:44.932163 (+ 5232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:44.933249 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:44.933264 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:44.933282 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:44.933565 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:44.934019 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:44.934024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:44.934028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:44.935086 (+ 1058us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:44.935488 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:44.936053 (+ 565us) server_negotiation.cc:299] Negotiation successful
0620 23:31:44.936901 (+ 848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":472,"threads_started":1}
I20260620 23:31:44.941395 12815 raft_consensus.cc:348] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:44.941958 12815 raft_consensus.cc:374] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:44.942137 12815 raft_consensus.cc:729] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6e87fd944847c3805dc44d8b8b3a40, State: Initialized, Role: FOLLOWER
I20260620 23:31:44.943002 12815 consensus_queue.cc:260] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:44.943382 12815 raft_consensus.cc:388] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:31:44.943568 12815 raft_consensus.cc:482] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:31:44.943785 12815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:44.946560 12815 raft_consensus.cc:504] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:44.947309 12815 leader_election.cc:302] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da6e87fd944847c3805dc44d8b8b3a40; no voters:
I20260620 23:31:44.948258 12815 leader_election.cc:288] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:31:44.948493 12820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:31:44.949939 12820 raft_consensus.cc:686] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 LEADER]: Becoming Leader. State: Replica: da6e87fd944847c3805dc44d8b8b3a40, State: Running, Role: LEADER
I20260620 23:31:44.950870 12820 consensus_queue.cc:237] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:44.952816 12815 sys_catalog.cc:564] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:31:44.954337 12822 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da6e87fd944847c3805dc44d8b8b3a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } } }
I20260620 23:31:44.954396 12821 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da6e87fd944847c3805dc44d8b8b3a40. Latest consensus state: current_term: 1 leader_uuid: "da6e87fd944847c3805dc44d8b8b3a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } } }
I20260620 23:31:44.955220 12821 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:44.955192 12822 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:44.957674 12827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:31:44.964025 12827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:31:44.965291 12815 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 23:31:44.973452 12827 catalog_manager.cc:1347] Generated new cluster ID: 31337950b69e4c2e9c5c6f1139ec78e8
I20260620 23:31:44.973592 12827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:31:45.003293 12827 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:31:45.005964 12827 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:31:45.025429 12827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Generated new TSK 0
I20260620 23:31:45.027384 12827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:31:45.830127 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34567
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:31:46 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 23:31:46.372447 12860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:46.372920 12860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:46.373149 12860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:46.382905 12860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:46.383100 12860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:46.383318 12860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:31:46.398790 12860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.96.212:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:46.401407 12860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:46.403898 12860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:46.429701 12870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:46.432052 12871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:46.432648 12873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:46.434468 12860 server_base.cc:1034] running on GCE node
I20260620 23:31:46.435408 12860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:46.436972 12860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:46.438333 12860 hybrid_clock.cc:648] HybridClock initialized: now 1781998306438229 us; error 107 us; skew 500 ppm
I20260620 23:31:46.438793 12860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:46 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:46.444068 12860 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:31:46.446727 12860 webserver.cc:460] Webserver started at http://127.0.96.193:41307/ using document root <none> and password file <none>
I20260620 23:31:46.447976 12860 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:46.448158 12860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:46.448596 12860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:31:46 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:46.469050 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "88ef007d0b6f48cea9296b96f251c2fa"
format_stamp: "Formatted at 2026-06-20 23:31:46 on dist-test-slave-v34r"
server_key: "663ae608c25ffe1190eb11a1ce865c16"
server_key_iv: "d51d7ded5df75c88841fabce5d1cb44a"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:46.470335 12860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "88ef007d0b6f48cea9296b96f251c2fa"
format_stamp: "Formatted at 2026-06-20 23:31:46 on dist-test-slave-v34r"
server_key: "663ae608c25ffe1190eb11a1ce865c16"
server_key_iv: "d51d7ded5df75c88841fabce5d1cb44a"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:46.478998 12860 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260620 23:31:46.520867 12880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:46.527318 12860 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.013s sys 0.002s
I20260620 23:31:46.527740 12860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "88ef007d0b6f48cea9296b96f251c2fa"
format_stamp: "Formatted at 2026-06-20 23:31:46 on dist-test-slave-v34r"
server_key: "663ae608c25ffe1190eb11a1ce865c16"
server_key_iv: "d51d7ded5df75c88841fabce5d1cb44a"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:46.528214 12860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:46.550451 12860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:46.559046 12860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:46.559636 12860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:46.561420 12860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:31:46.565227 12860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:31:46.565457 12860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:46.565732 12860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:31:46.565883 12860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:31:46 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:46.772410 12860 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:33711
I20260620 23:31:46.772421 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:33711 every 8 connection(s)
I20260620 23:31:46.774348 12860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:31:46.789902 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 12860
I20260620 23:31:46.790125 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:31:46.790889 387 external_mini_cluster.cc:1428] Setting key 4c10cc22e875d43bbac13b8be4ac763c
Jun 20 23:31:46 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:31:46.824291 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:46.779539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34567 (local address 127.0.96.193:42387)
0620 23:31:46.790667 (+ 11128us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:46.790753 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:46.796306 (+ 5553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:46.799510 (+ 3204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:46.799529 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:46.800347 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:46.801595 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:46.801634 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:46.803720 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:46.803727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:46.806162 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:46.806191 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:46.807806 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:46.807815 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:46.808184 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:46.809309 (+ 1125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:46.809356 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:46.812609 (+ 3253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:46.817757 (+ 5148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:46.817771 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:46.817826 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:46.818586 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:46.819158 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:46.819163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:46.819166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:46.819499 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:46.820537 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:46.820548 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:46.821355 (+ 807us) client_negotiation.cc:769] Sending connection context
0620 23:31:46.821765 (+ 410us) client_negotiation.cc:240] Negotiation successful
0620 23:31:46.822469 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10541,"thread_start_us":13795,"threads_started":1}
I20260620 23:31:46.825970 12995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:46.778631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:42387 (local address 127.0.96.254:34567)
0620 23:31:46.794274 (+ 15643us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:46.794281 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:46.794309 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:31:46.796712 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:46.796719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:46.796858 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:46.796972 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:46.801996 (+ 5024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:46.803401 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:46.806475 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:46.807155 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:46.807290 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:46.813113 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:46.813123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:46.813129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:46.813192 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:46.817393 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:46.818783 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:46.818790 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:46.818793 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:46.818985 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:46.819776 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:46.819786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:46.819790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:46.820320 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:46.820551 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:46.825187 (+ 4636us) server_negotiation.cc:299] Negotiation successful
0620 23:31:46.825517 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15488,"thread_start_us":15648,"threads_started":1}
I20260620 23:31:46.827849 12993 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34567
I20260620 23:31:46.828240 12993 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:46.829299 12993 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:46.832974 12780 ts_manager.cc:194] Registered new tserver with Master: 88ef007d0b6f48cea9296b96f251c2fa (127.0.96.193:33711)
I20260620 23:31:46.835831 12780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:42387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:31:46.860728 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34567
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:31:47 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
W20260620 23:31:47.397897 12999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:47.398480 12999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:47.398762 12999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:47.410975 12999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:47.411164 12999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:47.411304 12999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:31:47.423244 12999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.96.212:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:47.425091 12999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:47.426874 12999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:47.444396 13014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:47.444386 13013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:47.444694 13016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:47.445760 12999 server_base.cc:1034] running on GCE node
I20260620 23:31:47.446596 12999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:47.447944 12999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:47.449205 12999 hybrid_clock.cc:648] HybridClock initialized: now 1781998307449145 us; error 45 us; skew 500 ppm
I20260620 23:31:47.449633 12999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:47 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:47.453890 12999 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:31:47.455902 12999 webserver.cc:460] Webserver started at http://127.0.96.194:43527/ using document root <none> and password file <none>
I20260620 23:31:47.456871 12999 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:47.457063 12999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:47.457423 12999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:31:47 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:47.473439 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db"
format_stamp: "Formatted at 2026-06-20 23:31:47 on dist-test-slave-v34r"
server_key: "2599574082066fd699977e246eed3302"
server_key_iv: "71f23493f2fea75b9f2d150a84151cb4"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:47.474337 12999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db"
format_stamp: "Formatted at 2026-06-20 23:31:47 on dist-test-slave-v34r"
server_key: "2599574082066fd699977e246eed3302"
server_key_iv: "71f23493f2fea75b9f2d150a84151cb4"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:47.480302 12999 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:31:47.498255 13023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:47.500348 12999 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260620 23:31:47.500635 12999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db"
format_stamp: "Formatted at 2026-06-20 23:31:47 on dist-test-slave-v34r"
server_key: "2599574082066fd699977e246eed3302"
server_key_iv: "71f23493f2fea75b9f2d150a84151cb4"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:47.500952 12999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:47.529060 12999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:47.534607 12999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:47.535019 12999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:47.536369 12999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:31:47.538419 12999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:31:47.538506 12999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:47.538722 12999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:31:47.538777 12999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:47.580858 12999 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:43243
I20260620 23:31:47.580879 13135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:43243 every 8 connection(s)
I20260620 23:31:47.582620 12999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:31:47.590553 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 12999
I20260620 23:31:47.590819 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:31:47.591349 387 external_mini_cluster.cc:1428] Setting key 0fb37d6aa82c45fcb3bd540e44c71928
Jun 20 23:31:47 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:47.604874 13138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:47.585724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:44055 (local address 127.0.96.254:34567)
0620 23:31:47.586248 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:47.586253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:47.587617 (+ 1364us) server_negotiation.cc:407] Connection header received
0620 23:31:47.589587 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:47.589593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:47.589689 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:47.589812 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:47.591501 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:47.592450 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:47.594453 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:47.594865 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:47.595005 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:47.598492 (+ 3487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:47.598498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:47.598501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:47.598535 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:47.601716 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:47.602506 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:47.602511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:47.602517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:47.602650 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:47.603092 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:47.603103 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:47.603106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:47.603389 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:47.603548 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:47.604238 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 23:31:47.604466 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
I20260620 23:31:47.606048 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:47.586405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34567 (local address 127.0.96.194:44055)
0620 23:31:47.587420 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:47.587466 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:47.589295 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:47.590097 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:47.590105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:47.590578 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:47.591307 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:47.591329 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:47.592621 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:47.592625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:47.594253 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:47.594273 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:47.595106 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:47.595113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:47.595358 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:47.596091 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:47.596114 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:47.598247 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:47.601890 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:47.601899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:47.601923 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:47.602370 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:47.602761 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:47.602766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:47.602770 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:47.602972 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:47.603527 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:47.603535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:47.603874 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 23:31:47.604141 (+ 267us) client_negotiation.cc:240] Negotiation successful
0620 23:31:47.604671 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":468,"threads_started":1}
I20260620 23:31:47.607820 13136 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34567
I20260620 23:31:47.608179 13136 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:47.609074 13136 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:47.610810 12780 ts_manager.cc:194] Registered new tserver with Master: 1bbf8ad8ec6d44fe82a2953f6c7ba4db (127.0.96.194:43243)
I20260620 23:31:47.611671 12780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:44055
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:31:47.657407 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34567
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:31:47.842640 12993 heartbeater.cc:503] Master 127.0.96.254:34567 was elected leader, sending a full tablet report...
W20260620 23:31:48.015409 13142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:48.015714 13142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:48.015848 13142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:48.027684 13142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:48.027845 13142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:48.028084 13142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:31:48.039639 13142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.96.212:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:48.041064 13142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:48.042445 13142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:48.057595 13147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:48.057551 13148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:48.057642 13150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:48.058845 13142 server_base.cc:1034] running on GCE node
I20260620 23:31:48.059731 13142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:48.061117 13142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:48.062402 13142 hybrid_clock.cc:648] HybridClock initialized: now 1781998308062310 us; error 77 us; skew 500 ppm
I20260620 23:31:48.062906 13142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:48 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:48.067185 13142 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:31:48.069357 13142 webserver.cc:460] Webserver started at http://127.0.96.195:33677/ using document root <none> and password file <none>
I20260620 23:31:48.070313 13142 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:48.070492 13142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:48.070879 13142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 23:31:48 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:48.091903 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "b598295c018a41159323cf8f9f8212ba"
format_stamp: "Formatted at 2026-06-20 23:31:48 on dist-test-slave-v34r"
server_key: "ae463892bb8debd07c90ee11a1fc014e"
server_key_iv: "d729350475cb2df0b3d467afc5589037"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:48.093443 13142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "b598295c018a41159323cf8f9f8212ba"
format_stamp: "Formatted at 2026-06-20 23:31:48 on dist-test-slave-v34r"
server_key: "ae463892bb8debd07c90ee11a1fc014e"
server_key_iv: "d729350475cb2df0b3d467afc5589037"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:48.103125 13142 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20260620 23:31:48.125234 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:48.127682 13142 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:31:48.127879 13142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "b598295c018a41159323cf8f9f8212ba"
format_stamp: "Formatted at 2026-06-20 23:31:48 on dist-test-slave-v34r"
server_key: "ae463892bb8debd07c90ee11a1fc014e"
server_key_iv: "d729350475cb2df0b3d467afc5589037"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:48.128181 13142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:48.145203 13142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:48.150476 13142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:48.150810 13142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:48.151873 13142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:31:48.153645 13142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:31:48.153748 13142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:48.153899 13142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:31:48.153976 13142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:48.196568 13142 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:43303
I20260620 23:31:48.196630 13269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:43303 every 8 connection(s)
I20260620 23:31:48.198005 13142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:31:48.200357 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13142
I20260620 23:31:48.200587 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:31:48.201241 387 external_mini_cluster.cc:1428] Setting key 846c12b891a7c1fa56bac43b8bd62b64
Jun 20 23:31:48 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:48.225562 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:48.201741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38513 (local address 127.0.96.254:34567)
0620 23:31:48.202689 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:48.202695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:48.203958 (+ 1263us) server_negotiation.cc:407] Connection header received
0620 23:31:48.206968 (+ 3010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:48.206974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:48.207106 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:48.207240 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:48.209608 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.210433 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.212365 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.212843 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.213002 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:48.217654 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:48.217665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:48.217673 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:48.217730 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:48.221627 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.222718 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.222724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.222727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.222924 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.223553 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.223559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.223561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.223938 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:48.224085 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:48.224919 (+ 834us) server_negotiation.cc:299] Negotiation successful
0620 23:31:48.225260 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":550,"threads_started":1}
I20260620 23:31:48.226502 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:48.202537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34567 (local address 127.0.96.195:38513)
0620 23:31:48.203756 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:48.203841 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:48.206583 (+ 2742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:48.207703 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:48.207715 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:48.208381 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:48.209341 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:48.209370 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.210657 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.210662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:48.212161 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:48.212179 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.213167 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.213176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:48.213480 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:48.214361 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:48.214390 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:48.217242 (+ 2852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:48.221868 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:48.221877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:48.221901 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:48.222508 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:48.223073 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:48.223079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:48.223084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:48.223366 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:48.224097 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:48.224104 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:48.224506 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 23:31:48.224771 (+ 265us) client_negotiation.cc:240] Negotiation successful
0620 23:31:48.225304 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":493,"threads_started":1}
I20260620 23:31:48.228267 13270 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34567
I20260620 23:31:48.228729 13270 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:48.229990 13270 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:48.232149 12779 ts_manager.cc:194] Registered new tserver with Master: b598295c018a41159323cf8f9f8212ba (127.0.96.195:43303)
I20260620 23:31:48.233294 12779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:38513
I20260620 23:31:48.245923 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:31:48.247901 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 12860
Jun 20 23:31:48 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:31:48.268316 13275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:48.249233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.0.96.193:33711)
0620 23:31:48.250014 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:48.250022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:48.250058 (+ 36us) server_negotiation.cc:407] Connection header received
0620 23:31:48.250138 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:48.250146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:48.250455 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:48.250643 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:48.252232 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.253581 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.255556 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.256273 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.256446 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:48.261544 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:48.261571 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:48.261594 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:48.261653 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:48.265263 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.266131 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.266141 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.266143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.266348 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.266689 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.266692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.266694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.267092 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:48.267380 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:48.267691 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 23:31:48.267949 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":405,"threads_started":1}
I20260620 23:31:48.270040 12988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:48.364894 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 12860
I20260620 23:31:48.381628 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 12999
Jun 20 23:31:48 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:31:48.407251 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:48.383031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.0.96.194:43243)
0620 23:31:48.383712 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:48.383722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:48.383754 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:31:48.383842 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:48.383851 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:48.384131 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:48.384367 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:48.386054 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.387198 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.388702 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.389438 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.389599 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:48.400019 (+ 10420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:48.400040 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:48.400062 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:48.400133 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:48.403573 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.404559 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.404569 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.404572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.404741 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.405243 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.405249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.405252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.405697 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:48.406060 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:48.406419 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 23:31:48.406746 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":343,"threads_started":1}
I20260620 23:31:48.408829 13131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:48.494441 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 12999
I20260620 23:31:48.511826 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13142
Jun 20 23:31:48 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:31:48.530297 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:48.513339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33196 (local address 127.0.96.195:43303)
0620 23:31:48.514109 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:48.514117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:48.514149 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:31:48.514211 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:48.514217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:48.514443 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:48.514577 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:48.516155 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.517305 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.518758 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:48.519343 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:48.519451 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:48.524320 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:48.524332 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:48.524347 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:48.524388 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:48.527267 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.528066 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.528073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.528075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.528210 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:48.528554 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:48.528557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:48.528559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:48.528895 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:48.529456 (+ 561us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:48.529775 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 23:31:48.529988 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":381,"threads_started":1}
I20260620 23:31:48.531996 13265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:48.618794 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13142
I20260620 23:31:48.635480 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 12748
I20260620 23:31:48.637192 12810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:48.748728 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 12748
I20260620 23:31:48.773994 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34567
--webserver_interface=127.0.96.254
--webserver_port=34969
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34567
--ranger_config_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:31:49.110978 13282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:49.111303 13282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:49.111469 13282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:49.119141 13282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:31:49.119270 13282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:49.119316 13282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:49.119365 13282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:31:49.119462 13282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:31:49.130092 13282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34567
--ranger_config_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=34969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:49.131467 13282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:49.132802 13282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:49.147723 13287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:49.147650 13290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:49.147785 13288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:49.149169 13282 server_base.cc:1034] running on GCE node
I20260620 23:31:49.150101 13282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:49.151712 13282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:49.153035 13282 hybrid_clock.cc:648] HybridClock initialized: now 1781998309152868 us; error 112 us; skew 500 ppm
I20260620 23:31:49.153439 13282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:49 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:49.156800 13282 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:31:49.158507 13282 webserver.cc:460] Webserver started at http://127.0.96.254:34969/ using document root <none> and password file <none>
I20260620 23:31:49.159158 13282 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:49.159268 13282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:31:49 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:49.249539 13282 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:31:49.253356 13298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:49.255261 13282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:31:49.255453 13282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "da6e87fd944847c3805dc44d8b8b3a40"
format_stamp: "Formatted at 2026-06-20 23:31:44 on dist-test-slave-v34r"
server_key: "ed500d4dfcaef250267723ca571688f2"
server_key_iv: "913d332bc49bac96e9b1969e6ba366dd"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:49.256440 13282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:49.275179 13282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:49.279567 13282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:49.279819 13282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:49.300601 13282 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34567
I20260620 23:31:49.300648 13349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34567 every 8 connection(s)
I20260620 23:31:49.302194 13282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:31:49.304610 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13282
I20260620 23:31:49.305562 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:33711
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:34567
--webserver_port=41307
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:31:49.311235 13350 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:31:49.315637 13350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Bootstrap starting.
I20260620 23:31:49.339747 13350 log.cc:824] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:49.351230 13350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:31:49.351959 13350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Bootstrap complete.
I20260620 23:31:49.364508 13350 raft_consensus.cc:348] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:49.365844 13350 raft_consensus.cc:729] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da6e87fd944847c3805dc44d8b8b3a40, State: Initialized, Role: FOLLOWER
I20260620 23:31:49.366751 13350 consensus_queue.cc:260] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:49.367059 13350 raft_consensus.cc:388] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:31:49.367206 13350 raft_consensus.cc:482] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:31:49.367409 13350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:31:49.372403 13350 raft_consensus.cc:504] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:49.373183 13350 leader_election.cc:302] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da6e87fd944847c3805dc44d8b8b3a40; no voters:
I20260620 23:31:49.373973 13350 leader_election.cc:288] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:31:49.374143 13354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:31:49.375669 13354 raft_consensus.cc:686] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [term 2 LEADER]: Becoming Leader. State: Replica: da6e87fd944847c3805dc44d8b8b3a40, State: Running, Role: LEADER
I20260620 23:31:49.376398 13354 consensus_queue.cc:237] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } }
I20260620 23:31:49.377084 13350 sys_catalog.cc:564] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:31:49.379710 13356 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da6e87fd944847c3805dc44d8b8b3a40. Latest consensus state: current_term: 2 leader_uuid: "da6e87fd944847c3805dc44d8b8b3a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } } }
I20260620 23:31:49.380316 13356 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:49.380846 13355 sys_catalog.cc:455] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da6e87fd944847c3805dc44d8b8b3a40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da6e87fd944847c3805dc44d8b8b3a40" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34567 } } }
I20260620 23:31:49.381212 13355 sys_catalog.cc:458] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:49.384013 13361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:31:49.391968 13361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:31:49.393811 13361 catalog_manager.cc:1259] Loaded cluster ID: 31337950b69e4c2e9c5c6f1139ec78e8
I20260620 23:31:49.393951 13361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:31:49.398844 13361 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:31:49.401028 13361 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Loaded TSK: 0
I20260620 23:31:49.403272 13361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:31:49.407963 13350 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 23:31:49.784250 13352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:49.784721 13352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:49.784930 13352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:49.797159 13352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:49.797319 13352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:49.797508 13352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:31:49.813745 13352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:33711
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=41307
--tserver_master_addrs=127.0.96.254:34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:49.815495 13352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:49.817297 13352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:49.851056 13388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:49.859941 13387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:49.863157 13352 server_base.cc:1034] running on GCE node
W20260620 23:31:49.864219 13390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:49.865408 13352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:49.871169 13352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:49.874637 13352 hybrid_clock.cc:648] HybridClock initialized: now 1781998309874569 us; error 87 us; skew 500 ppm
I20260620 23:31:49.875198 13352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:49 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:49.883149 13352 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:31:49.901041 13352 webserver.cc:460] Webserver started at http://127.0.96.193:41307/ using document root <none> and password file <none>
I20260620 23:31:49.902261 13352 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:49.902535 13352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:31:49 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:49.949656 13352 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.009s sys 0.004s
I20260620 23:31:49.954276 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:49.956895 13352 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:31:49.957209 13352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "88ef007d0b6f48cea9296b96f251c2fa"
format_stamp: "Formatted at 2026-06-20 23:31:46 on dist-test-slave-v34r"
server_key: "663ae608c25ffe1190eb11a1ce865c16"
server_key_iv: "d51d7ded5df75c88841fabce5d1cb44a"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:49.959025 13352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:49.990069 13352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:49.996727 13352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:49.997176 13352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:49.998749 13352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:31:50.001809 13352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:31:50.002280 13352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:50.002476 13352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:31:50.002601 13352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:50.154556 13352 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:33711
I20260620 23:31:50.154589 13511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:33711 every 8 connection(s)
I20260620 23:31:50.157640 13352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:31:50.160924 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13352
I20260620 23:31:50.161734 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:43243
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:34567
--webserver_port=43527
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:31:50 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:50.204335 13515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:50.162576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34567 (local address 127.0.96.193:59633)
0620 23:31:50.174108 (+ 11532us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:50.174186 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:50.177080 (+ 2894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:50.180174 (+ 3094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:50.180189 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:50.181083 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:50.182050 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:50.182085 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.184960 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.184968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:50.187375 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:50.187396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.189601 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.189609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:50.189993 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:50.190966 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:50.191006 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:50.193773 (+ 2767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:50.198309 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:50.198323 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:50.198348 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:50.199018 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:50.199599 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:50.199605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:50.199609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:50.199945 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:50.200912 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:50.200921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:50.201406 (+ 485us) client_negotiation.cc:769] Sending connection context
0620 23:31:50.201728 (+ 322us) client_negotiation.cc:240] Negotiation successful
0620 23:31:50.202478 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11079,"thread_start_us":11079,"threads_started":1}
I20260620 23:31:50.206318 13514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:50.162382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:59633 (local address 127.0.96.254:34567)
0620 23:31:50.175409 (+ 13027us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x60c00040cb80. stack: 00007f349796b66e 00007f349796b430 00007f34969f25ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f34aa84e775 00007f34979b11d3 00007f34979b491b 00007f34979b4798 00000000009edb7c 00007f349798feae 00007f349ac696da 00007f3493bff61e
0620 23:31:50.175655 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:50.175678 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:50.175751 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:31:50.178268 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:50.178297 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:50.179011 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:50.179588 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:50.182427 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.184655 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.187602 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.188809 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.189079 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:50.194139 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:50.194152 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:50.194179 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:50.194265 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:50.198058 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:50.199224 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:50.199234 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:50.199240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:50.199458 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:50.200079 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:50.200084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:50.200090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:50.200750 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:50.201033 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:50.202438 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0620 23:31:50.202928 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10778,"spinlock_wait_cycles":3656064,"thread_start_us":10619,"threads_started":1}
I20260620 23:31:50.211699 13512 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34567
I20260620 23:31:50.212235 13512 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:50.213529 13512 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
W20260620 23:31:50.218677 13512 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:34567 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:31:50.219254 13512 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:50.219827 13512 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:50.221848 13512 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:50.222241 13512 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
W20260620 23:31:50.223783 13512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:31:50.617405 13516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:50.617692 13516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:50.617846 13516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:50.626307 13516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:50.626469 13516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:50.626657 13516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:31:50.641665 13516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:43243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=43527
--tserver_master_addrs=127.0.96.254:34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:50.643369 13516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:50.645087 13516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:50.662724 13531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:50.663427 13533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:50.664855 13530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:50.666175 13516 server_base.cc:1034] running on GCE node
I20260620 23:31:50.667037 13516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:50.668313 13516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:50.669826 13516 hybrid_clock.cc:648] HybridClock initialized: now 1781998310669737 us; error 88 us; skew 500 ppm
I20260620 23:31:50.670219 13516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:50 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:50.678277 13516 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:31:50.680169 13516 webserver.cc:460] Webserver started at http://127.0.96.194:43527/ using document root <none> and password file <none>
I20260620 23:31:50.680907 13516 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:50.681138 13516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:31:50 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:50.716930 13516 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.009s sys 0.008s
I20260620 23:31:50.739514 13541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:50.741546 13516 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.011s sys 0.012s
I20260620 23:31:50.741756 13516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db"
format_stamp: "Formatted at 2026-06-20 23:31:47 on dist-test-slave-v34r"
server_key: "2599574082066fd699977e246eed3302"
server_key_iv: "71f23493f2fea75b9f2d150a84151cb4"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:50.743175 13516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:50.765920 13516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:50.771678 13516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:50.771992 13516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:50.773393 13516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:31:50.775305 13516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:31:50.775441 13516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:50.775558 13516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:31:50.775635 13516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:31:50 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781998310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:50.853140 13516 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:43243
I20260620 23:31:50.853294 13655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:43243 every 8 connection(s)
I20260620 23:31:50.854883 13516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:31:50.862051 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13516
I20260620 23:31:50.862996 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.96.212:41959/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:43303
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:34567
--webserver_port=33677
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:31:50 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:50.894531 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:50.863219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:58105 (local address 127.0.96.254:34567)
0620 23:31:50.867892 (+ 4673us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:50.867900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:50.867924 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:31:50.868197 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:50.868203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:50.868307 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:50.868434 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:50.870771 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.872003 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.874476 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.875132 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.875235 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:50.880091 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:50.880099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:50.880104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:50.880153 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:50.886231 (+ 6078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:50.887410 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:50.887416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:50.887420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:50.887577 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:50.892056 (+ 4479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:50.892062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:50.892065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:50.892466 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:50.892621 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:50.893756 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0620 23:31:50.894036 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4528,"thread_start_us":4393,"threads_started":1}
I20260620 23:31:50.897224 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:50.862451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34567 (local address 127.0.96.194:58105)
0620 23:31:50.863814 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:50.863863 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:50.867128 (+ 3265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:50.868921 (+ 1793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:50.868934 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:50.869634 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:50.870533 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:50.870563 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.872218 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.872225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:50.874271 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:50.874288 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:50.875655 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:50.875662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:50.875945 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:50.876894 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:50.876927 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:50.879770 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:50.886470 (+ 6700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:50.886482 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:50.886504 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:50.887235 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:50.891535 (+ 4300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:50.891541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:50.891544 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:50.891888 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:50.892728 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:50.892737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:50.893273 (+ 536us) client_negotiation.cc:769] Sending connection context
0620 23:31:50.895119 (+ 1846us) client_negotiation.cc:240] Negotiation successful
0620 23:31:50.895720 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":1063,"threads_started":1}
I20260620 23:31:50.899829 13656 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34567
I20260620 23:31:50.900213 13656 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:50.901252 13656 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:50.903503 13315 ts_manager.cc:194] Registered new tserver with Master: 1bbf8ad8ec6d44fe82a2953f6c7ba4db (127.0.96.194:43243)
I20260620 23:31:50.905791 13315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:58105
I20260620 23:31:51.223979 13512 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:51.224619 13512 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:51.226437 13315 ts_manager.cc:194] Registered new tserver with Master: 88ef007d0b6f48cea9296b96f251c2fa (127.0.96.193:33711)
I20260620 23:31:51.227762 13315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:59633
Jun 20 23:31:51 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
W20260620 23:31:51.393718 13660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:51.394156 13660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:51.394407 13660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:51.406198 13660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:51.406348 13660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:51.406610 13660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:31:51.420720 13660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.96.212:41959/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:43303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=33677
--tserver_master_addrs=127.0.96.254:34567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:51.422274 13660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:51.423631 13660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:51.441804 13666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:51.442011 13667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:51.448125 13660 server_base.cc:1034] running on GCE node
W20260620 23:31:51.449030 13669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:51.449815 13660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:51.453073 13660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:51.454336 13660 hybrid_clock.cc:648] HybridClock initialized: now 1781998311454274 us; error 58 us; skew 500 ppm
I20260620 23:31:51.454710 13660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:51 dist-test-slave-v34r krb5kdc[12304](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:51.458082 13660 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:31:51.460387 13660 webserver.cc:460] Webserver started at http://127.0.96.195:33677/ using document root <none> and password file <none>
I20260620 23:31:51.461447 13660 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:51.461611 13660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 23:31:51 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:51.486730 13660 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:31:51.491948 13677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:51.494386 13660 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260620 23:31:51.494676 13660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "b598295c018a41159323cf8f9f8212ba"
format_stamp: "Formatted at 2026-06-20 23:31:48 on dist-test-slave-v34r"
server_key: "ae463892bb8debd07c90ee11a1fc014e"
server_key_iv: "d729350475cb2df0b3d467afc5589037"
server_key_version: "kuduclusterkey@0"
I20260620 23:31:51.496372 13660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:51.547401 13660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:51.554426 13660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:51.554832 13660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:51.556447 13660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:31:51.558768 13660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:31:51.558916 13660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:51.559051 13660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:31:51.559129 13660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 23:31:51 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for HTTP/127.0.96.212@KRBTEST.COM
I20260620 23:31:51.706434 13660 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:43303
I20260620 23:31:51.706522 13790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:43303 every 8 connection(s)
I20260620 23:31:51.709038 13660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:31:51.711436 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13660
Jun 20 23:31:51 dist-test-slave-v34r krb5kdc[12304](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:51.776225 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.717066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.0.96.254:34567)
0620 23:31:51.734654 (+ 17588us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:51.734666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:51.743826 (+ 9160us) server_negotiation.cc:407] Connection header received
0620 23:31:51.744087 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:51.744094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:51.744253 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:51.744464 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:51.758525 (+ 14061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.759874 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.762379 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.763160 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.763327 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:51.768104 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:51.768112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:51.768119 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:51.768188 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:51.772131 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:51.773524 (+ 1393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:51.773531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:51.773535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:51.773733 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:51.774290 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:51.774296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:51.774300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:51.774714 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:51.775039 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:51.775478 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 23:31:51.775744 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17330,"thread_start_us":27349,"threads_started":1}
I20260620 23:31:51.789906 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.726071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:53931 (local address 127.0.96.254:34567)
0620 23:31:51.744648 (+ 18577us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:51.744658 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:51.744686 (+ 28us) server_negotiation.cc:407] Connection header received
0620 23:31:51.744762 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:51.744768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:51.744871 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:51.745055 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:51.747704 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.749097 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.751977 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.752716 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.752857 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:51.764698 (+ 11841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:51.764705 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:51.764710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:51.764755 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:51.777997 (+ 13242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:51.784024 (+ 6027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:51.784031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:51.784035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:51.784258 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:51.785032 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:51.785038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:51.785042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:51.785476 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:51.785627 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:51.789195 (+ 3568us) server_negotiation.cc:299] Negotiation successful
0620 23:31:51.789479 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18385,"thread_start_us":8112,"threads_started":1}
I20260620 23:31:51.788913 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.725099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34567 (local address 127.0.96.195:53931)
0620 23:31:51.729921 (+ 4822us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:51.730005 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:51.733119 (+ 3114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:51.745528 (+ 12409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:51.745546 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:51.746364 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:51.747394 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:51.747427 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.749308 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.749314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:51.751712 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:51.751737 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.756336 (+ 4599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.756346 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:51.756748 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:51.760265 (+ 3517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:51.760307 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:51.764367 (+ 4060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:51.778228 (+ 13861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:51.778241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:51.778264 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:51.779028 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:51.784447 (+ 5419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:51.784453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:51.784457 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:51.784827 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:51.785732 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:51.785743 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:51.786282 (+ 539us) client_negotiation.cc:769] Sending connection context
0620 23:31:51.786656 (+ 374us) client_negotiation.cc:240] Negotiation successful
0620 23:31:51.787329 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4308,"thread_start_us":7782,"threads_started":1}
I20260620 23:31:51.792901 13791 heartbeater.cc:347] Connected to a master server at 127.0.96.254:34567
I20260620 23:31:51.793296 13791 heartbeater.cc:464] Registering TS with master...
I20260620 23:31:51.794334 13791 heartbeater.cc:511] Master 127.0.96.254:34567 requested a full tablet report, sending...
I20260620 23:31:51.797046 13315 ts_manager.cc:194] Registered new tserver with Master: b598295c018a41159323cf8f9f8212ba (127.0.96.195:43303)
I20260620 23:31:51.798370 13315 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:53931
I20260620 23:31:51.808595 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:31:51.899451 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.861407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.0.96.254:34567)
0620 23:31:51.863622 (+ 2215us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:51.863630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:51.875260 (+ 11630us) server_negotiation.cc:407] Connection header received
0620 23:31:51.875687 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:51.875695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:51.875812 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:51.875906 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:51.879288 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.880838 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.883315 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:51.884117 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:51.884267 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:51.890035 (+ 5768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:51.890046 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:51.890052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:51.890141 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:51.894719 (+ 4578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:51.896052 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:51.896062 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:51.896066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:51.896316 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:51.896954 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:51.896961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:51.896965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:51.897497 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:51.897820 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:51.898631 (+ 811us) server_negotiation.cc:299] Negotiation successful
0620 23:31:51.898983 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 23:31:51.910977 13656 heartbeater.cc:503] Master 127.0.96.254:34567 was elected leader, sending a full tablet report...
I20260620 23:31:51.909638 13315 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:31:51.927950 13315 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:31:52.021405 13810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.979242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:43243 (local address 127.0.0.1:59028)
0620 23:31:52.000478 (+ 21236us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:52.000554 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:52.000829 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:52.008778 (+ 7949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:52.008791 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:52.008856 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:31:52.009467 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.009486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.011757 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.011764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.014821 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.014840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.020145 (+ 5305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.020152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.020388 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.020459 (+ 71us) client_negotiation.cc:769] Sending connection context
0620 23:31:52.020635 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 23:31:52.020868 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20810,"spinlock_wait_cycles":2432,"thread_start_us":28031,"threads_started":1}
I20260620 23:31:52.022210 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.980234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.96.194:43243)
0620 23:31:52.005149 (+ 24915us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.005160 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.005202 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:31:52.005337 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.005347 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.005681 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.005990 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:31:52.009758 (+ 3768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.011539 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.015153 (+ 3614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.017107 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.017295 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.017634 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.020848 (+ 3214us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.021384 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24622,"thread_start_us":25965,"threads_started":1}
I20260620 23:31:52.028501 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.981713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:43303 (local address 127.0.0.1:33206)
0620 23:31:52.006991 (+ 25278us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:52.007019 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:52.007197 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:52.014559 (+ 7362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:52.014566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:52.014609 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:31:52.015154 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.015171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.017425 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.017430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.024191 (+ 6761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.024211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.027496 (+ 3285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.027504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.027720 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.027764 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:31:52.027868 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:31:52.028062 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25075,"thread_start_us":19311,"threads_started":1}
I20260620 23:31:52.029459 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.981380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:33711 (local address 127.0.0.1:41106)
0620 23:31:52.006363 (+ 24983us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:52.006391 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:52.006581 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:52.008775 (+ 2194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:52.008786 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:52.008864 (+ 78us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:31:52.009810 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.009831 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.014373 (+ 4542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.014380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.019790 (+ 5410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.019808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.027145 (+ 7337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.027157 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.027467 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.027515 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 23:31:52.027761 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 23:31:52.027981 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24816,"spinlock_wait_cycles":26112,"thread_start_us":25431,"threads_started":1}
I20260620 23:31:52.029505 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.982589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.0.96.195:43303)
0620 23:31:52.001917 (+ 19328us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.001934 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.007637 (+ 5703us) server_negotiation.cc:407] Connection header received
0620 23:31:52.007768 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.007779 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.008111 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.008354 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:31:52.015423 (+ 7069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.017202 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.025004 (+ 7802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.026776 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.026938 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.027228 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.028882 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.029131 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19014,"thread_start_us":21378,"threads_started":1}
I20260620 23:31:52.028921 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:51.981832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.0.96.193:33711)
0620 23:31:52.007632 (+ 25800us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.007644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.007687 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:31:52.007829 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.007840 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.008216 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.008524 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:31:52.012195 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.014043 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.024307 (+ 10264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.026293 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.026514 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.026890 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.028271 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.028504 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25555,"thread_start_us":31176,"threads_started":1}
I20260620 23:31:52.034672 13447 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6f570f4e6b4838b5388cec43f47f24 (DEFAULT_TABLE table=test-table [id=eb3f4b5f518347acb6fcfc4b4094864b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:31:52.034672 13726 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6f570f4e6b4838b5388cec43f47f24 (DEFAULT_TABLE table=test-table [id=eb3f4b5f518347acb6fcfc4b4094864b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:31:52.036763 13447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6f570f4e6b4838b5388cec43f47f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:52.036741 13726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6f570f4e6b4838b5388cec43f47f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:52.042824 13591 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6f570f4e6b4838b5388cec43f47f24 (DEFAULT_TABLE table=test-table [id=eb3f4b5f518347acb6fcfc4b4094864b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:31:52.044582 13591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6f570f4e6b4838b5388cec43f47f24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:52.067576 13816 tablet_bootstrap.cc:492] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Bootstrap starting.
I20260620 23:31:52.072005 13818 tablet_bootstrap.cc:492] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Bootstrap starting.
I20260620 23:31:52.074915 13816 tablet_bootstrap.cc:654] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:52.076290 13818 tablet_bootstrap.cc:654] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:52.077824 13818 log.cc:824] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:52.077953 13817 tablet_bootstrap.cc:492] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Bootstrap starting.
I20260620 23:31:52.082600 13818 tablet_bootstrap.cc:492] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: No bootstrap required, opened a new log
I20260620 23:31:52.082876 13817 tablet_bootstrap.cc:654] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:52.082985 13818 ts_tablet_manager.cc:1397] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 23:31:52.084177 13817 log.cc:824] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:52.085052 13816 log.cc:824] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:52.090741 13817 tablet_bootstrap.cc:492] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: No bootstrap required, opened a new log
I20260620 23:31:52.091332 13817 ts_tablet_manager.cc:1397] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260620 23:31:52.092799 13816 tablet_bootstrap.cc:492] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: No bootstrap required, opened a new log
I20260620 23:31:52.093246 13816 ts_tablet_manager.cc:1397] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.006s
I20260620 23:31:52.095438 13818 raft_consensus.cc:348] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.096042 13818 raft_consensus.cc:374] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:52.096166 13818 raft_consensus.cc:729] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbf8ad8ec6d44fe82a2953f6c7ba4db, State: Initialized, Role: FOLLOWER
I20260620 23:31:52.097048 13818 consensus_queue.cc:260] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.105765 13818 ts_tablet_manager.cc:1428] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Time spent starting tablet: real 0.023s user 0.011s sys 0.011s
I20260620 23:31:52.105548 13816 raft_consensus.cc:348] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.106230 13816 raft_consensus.cc:374] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:52.106356 13816 raft_consensus.cc:729] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b598295c018a41159323cf8f9f8212ba, State: Initialized, Role: FOLLOWER
I20260620 23:31:52.107291 13816 consensus_queue.cc:260] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.108747 13791 heartbeater.cc:503] Master 127.0.96.254:34567 was elected leader, sending a full tablet report...
I20260620 23:31:52.109017 13817 raft_consensus.cc:348] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.109661 13817 raft_consensus.cc:374] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:52.109763 13816 ts_tablet_manager.cc:1428] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260620 23:31:52.109833 13817 raft_consensus.cc:729] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ef007d0b6f48cea9296b96f251c2fa, State: Initialized, Role: FOLLOWER
I20260620 23:31:52.110772 13817 consensus_queue.cc:260] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.112464 13512 heartbeater.cc:503] Master 127.0.96.254:34567 was elected leader, sending a full tablet report...
I20260620 23:31:52.113270 13817 ts_tablet_manager.cc:1428] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Time spent starting tablet: real 0.022s user 0.016s sys 0.001s
W20260620 23:31:52.114393 13657 tablet.cc:2369] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:31:52.164111 13513 tablet.cc:2369] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:31:52.242537 13792 tablet.cc:2369] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:31:52.453675 13826 raft_consensus.cc:482] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:31:52.453976 13826 raft_consensus.cc:504] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.455705 13826 leader_election.cc:288] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88ef007d0b6f48cea9296b96f251c2fa (127.0.96.193:33711), b598295c018a41159323cf8f9f8212ba (127.0.96.195:43303)
I20260620 23:31:52.464305 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.456459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:58391 (local address 127.0.96.195:43303)
0620 23:31:52.456669 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.456676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.457724 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 23:31:52.457968 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.458005 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.458098 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.458237 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:31:52.459035 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.460229 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.462482 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.463325 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.463512 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.463604 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.463906 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.464040 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:31:52.464504 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.456182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:43303 (local address 127.0.96.194:58391)
0620 23:31:52.457487 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:52.457516 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:52.457750 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:52.458276 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:52.458281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:52.458317 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:31:52.458825 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.458838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.460406 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.460410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.462278 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.462292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.463556 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.463563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.463738 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.463764 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:31:52.463881 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:31:52.464030 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":20,"spinlock_wait_cycles":16384,"thread_start_us":1015,"threads_started":1}
I20260620 23:31:52.464768 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.456573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:49551 (local address 127.0.96.193:33711)
0620 23:31:52.456820 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.456827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.457723 (+ 896us) server_negotiation.cc:407] Connection header received
0620 23:31:52.457969 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.457976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.458098 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.458237 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:31:52.459493 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.460561 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.462769 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.463619 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.463846 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.463947 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.464300 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.464455 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:31:52.465171 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.456182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:33711 (local address 127.0.96.194:49551)
0620 23:31:52.457534 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:52.457554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:52.457750 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:52.458276 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:52.458281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:52.458319 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:31:52.459267 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.459278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.460748 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.460755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.462546 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:52.462566 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.463925 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.463932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:52.464113 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.464139 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:31:52.464264 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 23:31:52.464432 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":254,"spinlock_wait_cycles":28416,"thread_start_us":423,"threads_started":1}
I20260620 23:31:52.465811 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6f570f4e6b4838b5388cec43f47f24" candidate_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b598295c018a41159323cf8f9f8212ba" is_pre_election: true
I20260620 23:31:52.466325 13746 raft_consensus.cc:2454] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbf8ad8ec6d44fe82a2953f6c7ba4db in term 0.
I20260620 23:31:52.466423 13467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6f570f4e6b4838b5388cec43f47f24" candidate_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ef007d0b6f48cea9296b96f251c2fa" is_pre_election: true
I20260620 23:31:52.467020 13545 leader_election.cc:302] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbf8ad8ec6d44fe82a2953f6c7ba4db, b598295c018a41159323cf8f9f8212ba; no voters:
I20260620 23:31:52.467147 13467 raft_consensus.cc:2454] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bbf8ad8ec6d44fe82a2953f6c7ba4db in term 0.
I20260620 23:31:52.467443 13826 raft_consensus.cc:2781] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:31:52.467586 13826 raft_consensus.cc:482] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:31:52.467697 13826 raft_consensus.cc:3037] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:52.470033 13826 raft_consensus.cc:504] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.470718 13826 leader_election.cc:288] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [CANDIDATE]: Term 1 election: Requested vote from peers 88ef007d0b6f48cea9296b96f251c2fa (127.0.96.193:33711), b598295c018a41159323cf8f9f8212ba (127.0.96.195:43303)
I20260620 23:31:52.471405 13467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6f570f4e6b4838b5388cec43f47f24" candidate_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88ef007d0b6f48cea9296b96f251c2fa"
I20260620 23:31:52.471405 13746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6f570f4e6b4838b5388cec43f47f24" candidate_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b598295c018a41159323cf8f9f8212ba"
I20260620 23:31:52.471709 13467 raft_consensus.cc:3037] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:52.471698 13746 raft_consensus.cc:3037] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:52.475448 13467 raft_consensus.cc:2454] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbf8ad8ec6d44fe82a2953f6c7ba4db in term 1.
I20260620 23:31:52.475448 13746 raft_consensus.cc:2454] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bbf8ad8ec6d44fe82a2953f6c7ba4db in term 1.
I20260620 23:31:52.476050 13545 leader_election.cc:302] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bbf8ad8ec6d44fe82a2953f6c7ba4db, b598295c018a41159323cf8f9f8212ba; no voters:
I20260620 23:31:52.476429 13826 raft_consensus.cc:2781] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:31:52.477099 13826 raft_consensus.cc:686] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 1 LEADER]: Becoming Leader. State: Replica: 1bbf8ad8ec6d44fe82a2953f6c7ba4db, State: Running, Role: LEADER
I20260620 23:31:52.477875 13826 consensus_queue.cc:237] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } }
I20260620 23:31:52.483155 13314 catalog_manager.cc:5495] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bbf8ad8ec6d44fe82a2953f6c7ba4db (127.0.96.194). New cstate: current_term: 1 leader_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bbf8ad8ec6d44fe82a2953f6c7ba4db" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 } health_report { overall_health: UNKNOWN } } }
I20260620 23:31:52.586129 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.579359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.0.96.194:43243)
0620 23:31:52.580195 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.580199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.580237 (+ 38us) server_negotiation.cc:407] Connection header received
0620 23:31:52.580338 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.580342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.580430 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.580582 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:31:52.581332 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.582195 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.583785 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.584224 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.584324 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.584842 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:31:52.585516 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:31:52.585631 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.585777 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.585845 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":569,"threads_started":1}
I20260620 23:31:52.596230 13746 raft_consensus.cc:1261] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 1 FOLLOWER]: Refusing update from remote peer 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:31:52.596232 13467 raft_consensus.cc:1261] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 1 FOLLOWER]: Refusing update from remote peer 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:31:52.597733 13831 consensus_queue.cc:1035] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "88ef007d0b6f48cea9296b96f251c2fa" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:31:52.598155 13826 consensus_queue.cc:1035] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b598295c018a41159323cf8f9f8212ba" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 43303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:31:52.631518 13315 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:31:52.631923 13315 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:31:52.637051 13315 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da6e87fd944847c3805dc44d8b8b3a40: Sending DeleteTablet for 3 replicas of tablet 5d6f570f4e6b4838b5388cec43f47f24
I20260620 23:31:52.638782 13591 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6f570f4e6b4838b5388cec43f47f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:31:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:59028
I20260620 23:31:52.638775 13447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6f570f4e6b4838b5388cec43f47f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:31:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:41106
I20260620 23:31:52.638897 13726 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6f570f4e6b4838b5388cec43f47f24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:31:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:33206
I20260620 23:31:52.647059 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13352
I20260620 23:31:52.651048 13849 tablet_replica.cc:330] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: stopping tablet replica
I20260620 23:31:52.651957 13849 raft_consensus.cc:2229] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:31:52.653059 13849 raft_consensus.cc:2258] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:31:52.654163 13850 tablet_replica.cc:330] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: stopping tablet replica
I20260620 23:31:52.654314 13851 tablet_replica.cc:330] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: stopping tablet replica
I20260620 23:31:52.654703 13850 raft_consensus.cc:2229] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:31:52.655253 13851 raft_consensus.cc:2229] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:31:52.655326 13850 raft_consensus.cc:2258] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:31:52.656006 13851 raft_consensus.cc:2258] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:31:52.659196 13851 ts_tablet_manager.cc:1905] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:31:52.659222 13849 ts_tablet_manager.cc:1905] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:31:52.659611 13850 ts_tablet_manager.cc:1905] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:31:52.668365 13849 ts_tablet_manager.cc:1918] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:31:52.668541 13850 ts_tablet_manager.cc:1918] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:31:52.668749 13849 log.cc:1196] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/5d6f570f4e6b4838b5388cec43f47f24
I20260620 23:31:52.668856 13850 log.cc:1196] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/5d6f570f4e6b4838b5388cec43f47f24
I20260620 23:31:52.669543 13850 ts_tablet_manager.cc:1939] T 5d6f570f4e6b4838b5388cec43f47f24 P 88ef007d0b6f48cea9296b96f251c2fa: Deleting consensus metadata
I20260620 23:31:52.669701 13849 ts_tablet_manager.cc:1939] T 5d6f570f4e6b4838b5388cec43f47f24 P 1bbf8ad8ec6d44fe82a2953f6c7ba4db: Deleting consensus metadata
I20260620 23:31:52.671944 13302 catalog_manager.cc:4841] TS 88ef007d0b6f48cea9296b96f251c2fa (127.0.96.193:33711): tablet 5d6f570f4e6b4838b5388cec43f47f24 (table test-table [id=eb3f4b5f518347acb6fcfc4b4094864b]) successfully deleted
I20260620 23:31:52.672056 13301 catalog_manager.cc:4841] TS 1bbf8ad8ec6d44fe82a2953f6c7ba4db (127.0.96.194:43243): tablet 5d6f570f4e6b4838b5388cec43f47f24 (table test-table [id=eb3f4b5f518347acb6fcfc4b4094864b]) successfully deleted
I20260620 23:31:52.673605 13851 ts_tablet_manager.cc:1918] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:31:52.673956 13851 log.cc:1196] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/5d6f570f4e6b4838b5388cec43f47f24
I20260620 23:31:52.674618 13851 ts_tablet_manager.cc:1939] T 5d6f570f4e6b4838b5388cec43f47f24 P b598295c018a41159323cf8f9f8212ba: Deleting consensus metadata
I20260620 23:31:52.678246 13300 catalog_manager.cc:4841] TS b598295c018a41159323cf8f9f8212ba (127.0.96.195:43303): tablet 5d6f570f4e6b4838b5388cec43f47f24 (table test-table [id=eb3f4b5f518347acb6fcfc4b4094864b]) successfully deleted
I20260620 23:31:52.687850 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.649346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.96.193:33711)
0620 23:31:52.649637 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.649649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.657030 (+ 7381us) server_negotiation.cc:407] Connection header received
0620 23:31:52.657400 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.657406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.657512 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.657634 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:52.662798 (+ 5164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.663998 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.672684 (+ 8686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.673569 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.673750 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.678144 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:52.678163 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:52.678186 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:52.678245 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:52.682887 (+ 4642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:52.684292 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:52.684311 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:52.684319 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:52.684594 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:52.685406 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:52.685412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:52.685416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:52.686098 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:52.686518 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.687142 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.687430 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:31:52.688737 13507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:52.792304 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13352
I20260620 23:31:52.813206 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13516
I20260620 23:31:52.831983 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.814206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.96.194:43243)
0620 23:31:52.814466 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.814475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.814502 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:31:52.814586 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.814592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.814693 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.814843 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:52.816290 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.817676 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.819109 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.819892 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.820049 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.823211 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:52.823232 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:52.823256 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:52.823306 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:52.828039 (+ 4733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:52.829210 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:52.829221 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:52.829225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:52.829408 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:52.830065 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:52.830072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:52.830078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:52.830591 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:52.830913 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:52.831285 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 23:31:52.831603 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:31:52.832875 13651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:52.964622 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13516
I20260620 23:31:52.985769 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13660
I20260620 23:31:53.002123 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:52.986733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.0.96.195:43303)
0620 23:31:52.987323 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:52.987327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:52.987349 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:31:52.987419 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:52.987423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:52.987502 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:52.987632 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:52.989263 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.990202 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.991707 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:52.992192 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:52.992306 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:52.995440 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:52.995455 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:52.995472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:52.995514 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:52.998804 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:52.999857 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:52.999864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:52.999866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:53.000037 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:53.000539 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:53.000542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:53.000545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:53.000952 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:53.001321 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:53.001664 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:31:53.001836 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":337,"threads_started":1}
I20260620 23:31:53.003120 13786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:53.103816 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13660
I20260620 23:31:53.123519 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13282
I20260620 23:31:53.124960 13345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:53.259760 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13282
I20260620 23:31:53.284392 387 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 23:31:53.622164 387 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 23:31:53.622438 387 mini_ranger.cc:66] Stopping Ranger...
I20260620 23:31:53.968567 387 mini_ranger.cc:68] Stopped Ranger
2026-06-20 23:31:53.968 UTC [12334] LOG: received smart shutdown request
2026-06-20 23:31:53.973 UTC [12334] LOG: background worker "logical replication launcher" (PID 12343) exited with exit code 1
2026-06-20 23:31:53.973 UTC [12338] LOG: shutting down
2026-06-20 23:31:54.163 UTC [12334] LOG: database system is shut down
2026-06-20T23:31:54Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (60023 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:31:54 dist-test-slave-v34r krb5kdc[13866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:31:54 dist-test-slave-v34r krb5kdc[13866](info): set up 2 sockets
Jun 20 23:31:54 dist-test-slave-v34r krb5kdc[13866](info): commencing operation
krb5kdc: starting...
W20260620 23:31:55.425414 387 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:31:55 dist-test-slave-v34r krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:31:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:31:55.697994 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34319
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34319,127.0.96.253:40383,127.0.96.252:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:31:56.059696 13879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:56.060011 13879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:56.060171 13879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:56.068769 13879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:31:56.068917 13879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:56.069041 13879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:56.069159 13879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:31:56.069234 13879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:31:56.079901 13879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34319,127.0.96.253:40383,127.0.96.252:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:56.081663 13879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:56.083106 13879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:56.098140 13884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:56.098280 13885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:56.098287 13887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:56.098615 13879 server_base.cc:1034] running on GCE node
I20260620 23:31:56.099999 13879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:56.102104 13879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:56.103436 13879 hybrid_clock.cc:648] HybridClock initialized: now 1781998316103357 us; error 80 us; skew 500 ppm
I20260620 23:31:56.103927 13879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:56 dist-test-slave-v34r krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:56.111366 13879 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:31:56.113564 13879 webserver.cc:460] Webserver started at http://127.0.96.254:41449/ using document root <none> and password file <none>
I20260620 23:31:56.114524 13879 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:56.114712 13879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:56.115100 13879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:31:56.119232 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "2feea8db8b4842c1b1919b8cf85bf806"
format_stamp: "Formatted at 2026-06-20 23:31:56 on dist-test-slave-v34r"
server_key: "f67453dd51523909284f665afafccd06"
server_key_iv: "3300d06c77840846fdd5c4f07e99a594"
server_key_version: "encryptionkey@0"
I20260620 23:31:56.120374 13879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "2feea8db8b4842c1b1919b8cf85bf806"
format_stamp: "Formatted at 2026-06-20 23:31:56 on dist-test-slave-v34r"
server_key: "f67453dd51523909284f665afafccd06"
server_key_iv: "3300d06c77840846fdd5c4f07e99a594"
server_key_version: "encryptionkey@0"
I20260620 23:31:56.127919 13879 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.006s
I20260620 23:31:56.133334 13894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:56.135392 13879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:31:56.135702 13879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "2feea8db8b4842c1b1919b8cf85bf806"
format_stamp: "Formatted at 2026-06-20 23:31:56 on dist-test-slave-v34r"
server_key: "f67453dd51523909284f665afafccd06"
server_key_iv: "3300d06c77840846fdd5c4f07e99a594"
server_key_version: "encryptionkey@0"
I20260620 23:31:56.136008 13879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:56.164253 13879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:56.169656 13879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:56.170032 13879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:56.193749 13879 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34319
I20260620 23:31:56.193773 13945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34319 every 8 connection(s)
I20260620 23:31:56.195573 13879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:31:56.197424 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13879
I20260620 23:31:56.197682 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:31:56.198599 387 external_mini_cluster.cc:1428] Setting key dc5e79f77b78132302654c70d0d6e72c
I20260620 23:31:56.201016 13946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:56.210249 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.253@KRBTEST.COM" created.
Jun 20 23:31:56 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:56.241413 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.213886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.0.96.254:34319)
0620 23:31:56.215071 (+ 1185us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:56.215094 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:56.218001 (+ 2907us) server_negotiation.cc:407] Connection header received
0620 23:31:56.220411 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:56.220424 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:56.220932 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:56.221316 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:56.222837 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.224621 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.226571 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.227268 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.227452 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.232354 (+ 4902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:56.232369 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:56.232391 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:56.232449 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:56.235881 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.237172 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.237180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.237183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.237379 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.237936 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.237942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.237945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.238552 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:56.238729 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:56.239580 (+ 851us) server_negotiation.cc:299] Negotiation successful
0620 23:31:56.239946 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":859,"thread_start_us":542,"threads_started":1}
I20260620 23:31:56.241463 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.213886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34319 (local address 127.0.0.1:37446)
0620 23:31:56.217719 (+ 3833us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.217777 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:56.219791 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:56.221363 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:56.221370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:56.222075 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:56.222547 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.222567 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.224845 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.224850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.226382 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.226397 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.227580 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.227590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.227843 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.228716 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:56.228752 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:56.232011 (+ 3259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:56.236111 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.236134 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.236154 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.236949 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.237517 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.237525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.237533 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.237792 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.238733 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:56.238745 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:56.239103 (+ 358us) client_negotiation.cc:769] Sending connection context
0620 23:31:56.239434 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 23:31:56.240194 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3521,"mutex_wait_us":81,"spinlock_wait_cycles":63872,"thread_start_us":3261,"threads_started":1}
I20260620 23:31:56.245445 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:31:56.246987 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.246281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:45684)
0620 23:31:56.246551 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.246744 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260620 23:31:56.247414 13895 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
I20260620 23:31:56.249552 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.248858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:45694)
0620 23:31:56.249146 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.249295 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 23:31:56.249845 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:40383: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab.
Entry for principal kudu/127.0.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab.
I20260620 23:31:56.261257 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:40383
--webserver_interface=127.0.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab
--principal=kudu/127.0.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34319,127.0.96.253:40383,127.0.96.252:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:31:56.299458 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } attempt: 1
I20260620 23:31:56.301231 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.300451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:45708)
0620 23:31:56.300768 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.300974 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 23:31:56.302819 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.302248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:45710)
0620 23:31:56.302441 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.302559 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 23:31:56.303099 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:40383: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
I20260620 23:31:56.371418 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } attempt: 2
I20260620 23:31:56.373102 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.372163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37500)
0620 23:31:56.372546 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.372754 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 23:31:56.374764 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.374145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37510)
0620 23:31:56.374399 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.374520 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 23:31:56.375131 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:40383: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
I20260620 23:31:56.466449 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } attempt: 3
I20260620 23:31:56.468123 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.467391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37512)
0620 23:31:56.467739 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.467904 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 23:31:56.469641 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.469131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37526)
0620 23:31:56.469333 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.469464 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:31:56.469875 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:40383: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
I20260620 23:31:56.613150 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } attempt: 4
I20260620 23:31:56.614713 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.613960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37528)
0620 23:31:56.614395 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.614549 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260620 23:31:56.615948 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.615516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37544)
0620 23:31:56.615677 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.615805 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:31:56.616185 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:40383: Network error: Client connection negotiation failed: client connection to 127.0.96.253:40383: connect: Connection refused (error 111)
W20260620 23:31:56.618119 13953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:56.618400 13953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:56.618491 13953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:31:56.625821 13953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:31:56.625941 13953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:56.626031 13953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:56.626108 13953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:31:56.626179 13953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:31:56.636384 13953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34319,127.0.96.253:40383,127.0.96.252:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab
--principal=kudu/127.0.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:40383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:56.637773 13953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:56.639088 13953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:31:56.653043 13961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:56.653172 13958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:56.653113 13959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:56.654120 13953 server_base.cc:1034] running on GCE node
I20260620 23:31:56.655119 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:56.656754 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:56.658095 13953 hybrid_clock.cc:648] HybridClock initialized: now 1781998316658057 us; error 80 us; skew 500 ppm
I20260620 23:31:56.658505 13953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:56 dist-test-slave-v34r krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:56.662111 13953 init.cc:377] Logged in from keytab as kudu/127.0.96.253@KRBTEST.COM (short username kudu)
I20260620 23:31:56.664039 13953 webserver.cc:460] Webserver started at http://127.0.96.253:36235/ using document root <none> and password file <none>
I20260620 23:31:56.664877 13953 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:56.665060 13953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:56.665393 13953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:31:56.668596 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data/instance:
uuid: "b9bb0ef050144baeb09417754b9b0604"
format_stamp: "Formatted at 2026-06-20 23:31:56 on dist-test-slave-v34r"
server_key: "ec127cb59c63ed9c144cea9deffacda0"
server_key_iv: "69793a138fb13058417a57b59a193787"
server_key_version: "encryptionkey@0"
I20260620 23:31:56.669487 13953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal/instance:
uuid: "b9bb0ef050144baeb09417754b9b0604"
format_stamp: "Formatted at 2026-06-20 23:31:56 on dist-test-slave-v34r"
server_key: "ec127cb59c63ed9c144cea9deffacda0"
server_key_iv: "69793a138fb13058417a57b59a193787"
server_key_version: "encryptionkey@0"
I20260620 23:31:56.675285 13953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 23:31:56.679673 13968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:56.681653 13953 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:31:56.681893 13953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal
uuid: "b9bb0ef050144baeb09417754b9b0604"
format_stamp: "Formatted at 2026-06-20 23:31:56 on dist-test-slave-v34r"
server_key: "ec127cb59c63ed9c144cea9deffacda0"
server_key_iv: "69793a138fb13058417a57b59a193787"
server_key_version: "encryptionkey@0"
I20260620 23:31:56.682161 13953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:56.732406 13953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:56.737020 13953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:56.737337 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:56.759155 13953 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.253:40383
I20260620 23:31:56.759248 14019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.253:40383 every 8 connection(s)
I20260620 23:31:56.760622 13953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
I20260620 23:31:56.765542 14020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:56.769042 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 13953
I20260620 23:31:56.769261 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-1/wal/instance
I20260620 23:31:56.769704 387 external_mini_cluster.cc:1428] Setting key c638569fb649c7b63e66c0b7c5d0e78a
I20260620 23:31:56.774065 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.252@KRBTEST.COM" created.
Jun 20 23:31:56 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:56.800750 14023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.777541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.0.96.254:34319)
0620 23:31:56.778195 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:56.778204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:56.779481 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 23:31:56.781821 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:56.781827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:56.781939 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:56.782079 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:56.783999 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.784824 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.786907 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.787357 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.787508 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.793747 (+ 6239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:56.793754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:56.793757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:56.793820 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:56.796935 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.798021 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.798033 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.798038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.798167 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.798644 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.798656 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.798658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.798971 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:56.799153 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:56.800055 (+ 902us) server_negotiation.cc:299] Negotiation successful
0620 23:31:56.800322 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":358,"threads_started":1}
I20260620 23:31:56.802106 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.778253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34319 (local address 127.0.0.1:52810)
0620 23:31:56.779257 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.779314 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:56.781458 (+ 2144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:56.782393 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:56.782404 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:56.783051 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:56.783789 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.783809 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.785049 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.785059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.786666 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.786680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.787634 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.787641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.787907 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.788802 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:56.788830 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:56.793354 (+ 4524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:56.797237 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.797250 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.797283 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.797855 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.798296 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.798303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.798307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.798518 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.799127 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:56.799136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:56.799632 (+ 496us) client_negotiation.cc:769] Sending connection context
0620 23:31:56.799956 (+ 324us) client_negotiation.cc:240] Negotiation successful
0620 23:31:56.800596 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":431,"threads_started":1}
I20260620 23:31:56.804682 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:31:56 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab.
Entry for principal kudu/127.0.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab.
I20260620 23:31:56.818253 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:38569
--webserver_interface=127.0.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab
--principal=kudu/127.0.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34319,127.0.96.253:40383,127.0.96.252:38569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:31:56.822026 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.805787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37558)
0620 23:31:56.806027 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.806049 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:56.806183 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:56.807444 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:56.807448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:56.807989 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:56.808146 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.808155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.809696 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.809705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.810607 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.810620 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.811592 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.811598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.811795 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.812518 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:56.812538 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:56.814923 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:56.819060 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.819066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.819071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.819504 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.819956 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.819960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.819964 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.820114 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.820804 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:56.820812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:56.820967 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:31:56.821183 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 23:31:56.821551 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 23:31:56.822084 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.806101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.0.96.253:40383)
0620 23:31:56.806926 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:56.806933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:56.806964 (+ 31us) server_negotiation.cc:407] Connection header received
0620 23:31:56.807015 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:56.807022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:56.807270 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:56.807461 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:56.808320 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.809554 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.810765 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.811406 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.811570 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.815177 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:56.815194 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:56.815214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:56.815252 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:56.818874 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.819669 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.819678 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.819680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.819819 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.820231 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.820238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.820242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.820674 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:56.820924 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:56.821351 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 23:31:56.821676 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":653,"thread_start_us":474,"threads_started":1}
I20260620 23:31:56.824961 14020 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:31:56.826619 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.825865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50398)
0620 23:31:56.826134 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.826347 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260620 23:31:56.827066 13970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:56.828888 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.828499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50414)
0620 23:31:56.828660 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.828756 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:31:56.829198 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:56.878597 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 1
I20260620 23:31:56.880075 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.879484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50418)
0620 23:31:56.879730 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.879865 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:31:56.881470 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.881071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50430)
0620 23:31:56.881243 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.881332 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:31:56.881839 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:56.915469 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } attempt: 5
Jun 20 23:31:56 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:31:56.932502 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.916452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37566)
0620 23:31:56.916697 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.916722 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:56.916934 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:56.917543 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:56.917547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:56.918028 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:56.918245 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.918257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.919747 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.919752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.920760 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:56.920774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.921844 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.921849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:56.922001 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.922855 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:56.922877 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:56.925574 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:56.929781 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.929786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.929789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.930300 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.930762 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:56.930766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:56.930768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:56.930923 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:56.931574 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:56.931578 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:56.931744 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 23:31:56.931926 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 23:31:56.932198 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 23:31:56.932657 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.916589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.0.96.253:40383)
0620 23:31:56.916859 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:56.916865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:56.916897 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:31:56.917215 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:56.917223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:56.917342 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:56.917494 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:56.918421 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.919520 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.920963 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:56.921644 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:56.921780 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:56.925859 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:56.925866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:56.925871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:56.925921 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:56.929537 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.930454 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.930459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.930463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.930634 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:56.931049 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:56.931054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:56.931057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:56.931446 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:56.931608 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:56.932080 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 23:31:56.932322 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 23:31:56.934099 13946 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:31:56.935503 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.934987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50436)
0620 23:31:56.935207 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.935309 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:31:56.936738 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.936361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50448)
0620 23:31:56.936525 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.936618 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 23:31:56.936970 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:56.950130 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 2
I20260620 23:31:56.951740 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.951144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50452)
0620 23:31:56.951408 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.951539 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 23:31:56.953305 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.952664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50462)
0620 23:31:56.952844 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.953147 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:31:56.953567 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:56.988296 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 1
I20260620 23:31:56.989853 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.989255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50468)
0620 23:31:56.989493 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.989633 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:31:56.991415 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:56.990968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50472)
0620 23:31:56.991164 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:56.991271 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 23:31:56.991680 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:57.044835 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 3
I20260620 23:31:57.046555 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.045826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50486)
0620 23:31:57.046102 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.046218 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 23:31:57.047973 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.047416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50492)
0620 23:31:57.047711 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.047808 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260620 23:31:57.048224 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:57.059949 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 2
I20260620 23:31:57.061471 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.060836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50496)
0620 23:31:57.061152 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.061255 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 23:31:57.062945 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.062464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50504)
0620 23:31:57.062667 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.062790 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260620 23:31:57.063217 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
W20260620 23:31:57.163599 14027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:31:57.163921 14027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:31:57.164022 14027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 23:31:57.169481 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 3
I20260620 23:31:57.171108 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.170429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50516)
0620 23:31:57.170739 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.170874 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 23:31:57.171871 14027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:31:57.172036 14027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:31:57.172132 14027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:31:57.172227 14027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:31:57.172340 14027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:31:57.172677 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.172193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50532)
0620 23:31:57.172441 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.172538 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260620 23:31:57.172947 13946 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
I20260620 23:31:57.182915 14027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34319,127.0.96.253:40383,127.0.96.252:38569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab
--principal=kudu/127.0.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:31:57.184234 14027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:31:57.185629 14027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 23:31:57.191476 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 4
I20260620 23:31:57.193557 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.192452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50540)
0620 23:31:57.193202 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.193338 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570}
I20260620 23:31:57.195122 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.194577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50554)
0620 23:31:57.194861 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.194956 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260620 23:31:57.195529 14020 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:38569: Network error: Client connection negotiation failed: client connection to 127.0.96.252:38569: connect: Connection refused (error 111)
W20260620 23:31:57.201571 14034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:57.201591 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:31:57.206625 14036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:31:57.207190 14027 server_base.cc:1034] running on GCE node
I20260620 23:31:57.208196 14027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:31:57.209947 14027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:31:57.211239 14027 hybrid_clock.cc:648] HybridClock initialized: now 1781998317211164 us; error 84 us; skew 500 ppm
I20260620 23:31:57.211651 14027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:31:57.215480 14027 init.cc:377] Logged in from keytab as kudu/127.0.96.252@KRBTEST.COM (short username kudu)
I20260620 23:31:57.217419 14027 webserver.cc:460] Webserver started at http://127.0.96.252:45107/ using document root <none> and password file <none>
I20260620 23:31:57.218220 14027 fs_manager.cc:359] Metadata directory not provided
I20260620 23:31:57.218375 14027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:31:57.218698 14027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:31:57.221858 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data/instance:
uuid: "62e6b5c733d94b27a1a9e385d204e06f"
format_stamp: "Formatted at 2026-06-20 23:31:57 on dist-test-slave-v34r"
server_key: "5db173477386160512934d2c37509b90"
server_key_iv: "8532fa8401c29a99b1502ebeafb93469"
server_key_version: "encryptionkey@0"
I20260620 23:31:57.222714 14027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal/instance:
uuid: "62e6b5c733d94b27a1a9e385d204e06f"
format_stamp: "Formatted at 2026-06-20 23:31:57 on dist-test-slave-v34r"
server_key: "5db173477386160512934d2c37509b90"
server_key_iv: "8532fa8401c29a99b1502ebeafb93469"
server_key_version: "encryptionkey@0"
I20260620 23:31:57.228950 14027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 23:31:57.233405 14043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:31:57.235280 14027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 23:31:57.235534 14027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal
uuid: "62e6b5c733d94b27a1a9e385d204e06f"
format_stamp: "Formatted at 2026-06-20 23:31:57 on dist-test-slave-v34r"
server_key: "5db173477386160512934d2c37509b90"
server_key_iv: "8532fa8401c29a99b1502ebeafb93469"
server_key_version: "encryptionkey@0"
I20260620 23:31:57.235767 14027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:31:57.261628 14027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:31:57.266276 14027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:31:57.266588 14027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:31:57.288473 14027 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.252:38569
I20260620 23:31:57.288584 14094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.252:38569 every 8 connection(s)
I20260620 23:31:57.290002 14027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
I20260620 23:31:57.294919 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14027
I20260620 23:31:57.295161 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781998087756853-387-0/minicluster-data/master-2/wal/instance
I20260620 23:31:57.295373 14095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:31:57.295735 387 external_mini_cluster.cc:1428] Setting key 779b596d59ac3c2f38b967061d7ab1ba
I20260620 23:31:57.307936 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:31:57.325320 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.298219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.0.96.252:38569)
0620 23:31:57.299395 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.299415 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.299489 (+ 74us) server_negotiation.cc:407] Connection header received
0620 23:31:57.301855 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.301898 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.302663 (+ 765us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.303301 (+ 638us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.304846 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.306979 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.308480 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.309691 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.309923 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.315290 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.315305 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.315331 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.315391 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.319614 (+ 4223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.320512 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.320523 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.320534 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.320844 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.321340 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.321345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.321348 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.322273 (+ 925us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.322524 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.323220 (+ 696us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.323819 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":521,"threads_started":1}
I20260620 23:31:57.330791 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.311551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:34319 (local address 127.0.0.1:52816)
0620 23:31:57.312916 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.313039 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:57.313252 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:57.313779 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:57.313790 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:57.314736 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:57.315346 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.315360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.316745 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.316751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.318790 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.318812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.319637 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.319644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.319862 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.320561 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:57.320586 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:57.323877 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:57.327486 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.327497 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.327504 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.328107 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.328660 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.328666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.328669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.328838 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.329616 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:57.329626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:57.329894 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 23:31:57.330090 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 23:31:57.330377 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":764,"threads_started":1}
I20260620 23:31:57.331171 14099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.311619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.0.96.254:34319)
0620 23:31:57.312443 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.312450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.313372 (+ 922us) server_negotiation.cc:407] Connection header received
0620 23:31:57.313441 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.313444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.313514 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.313602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.315615 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.316542 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.318992 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.319406 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.319511 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.324115 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.324121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.324124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.324161 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.327315 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.328314 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.328320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.328323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.328501 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.328962 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.328967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.328971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.329337 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.329496 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.330341 (+ 845us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.330680 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":349,"threads_started":1}
I20260620 23:31:57.332928 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:31:57.349123 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.329477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52830 (local address 127.0.96.254:34319)
0620 23:31:57.330253 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.330259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.330284 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:31:57.330340 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.330344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.330419 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.330547 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.332281 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.333487 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.335213 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.335809 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.335965 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.341727 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.341739 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.341746 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.341810 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.345378 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.346428 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.346433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.346436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.346619 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.347088 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.347095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.347101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.347548 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.347855 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.348260 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.348549 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":322,"threads_started":1}
I20260620 23:31:57.349951 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.333830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:40383 (local address 127.0.0.1:37570)
0620 23:31:57.334069 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.334089 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:57.334192 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:57.334582 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:57.334585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:57.334998 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:57.335250 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.335263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.336631 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.336638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.337997 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.338017 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.339231 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.339236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.339508 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.340376 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:57.340408 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:57.343067 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:57.346892 (+ 3825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.346898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.346901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.347559 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.348060 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.348064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.348066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.348300 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.348947 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:57.348952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:57.349202 (+ 250us) client_negotiation.cc:769] Sending connection context
0620 23:31:57.349389 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 23:31:57.349686 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:31:57.350236 13946 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 4
I20260620 23:31:57.351282 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.333905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.0.96.253:40383)
0620 23:31:57.334235 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.334242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.334266 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:31:57.334333 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.334338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.334425 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.334546 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.335435 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.336399 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.338223 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.338967 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.339147 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.343293 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.343298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.343302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.343339 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.346663 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.347723 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.347730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.347733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.347917 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.348466 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.348472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.348476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.348809 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.348944 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.350103 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.350840 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:31:57.352731 14095 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:31:57.369781 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.350883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50572)
0620 23:31:57.351151 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.351171 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:57.351289 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:57.352074 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:57.352079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:57.352500 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:57.352727 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.352738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.354535 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.354541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.356031 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.356048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.357291 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.357297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.357497 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.358335 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:57.358364 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:57.361113 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:57.366197 (+ 5084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.366202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.366207 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.366782 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.367354 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.367363 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.367368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.367577 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.368380 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:57.368390 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:57.368651 (+ 261us) client_negotiation.cc:769] Sending connection context
0620 23:31:57.368846 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 23:31:57.369314 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260620 23:31:57.369879 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.351534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.0.96.252:38569)
0620 23:31:57.351730 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.351736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.351755 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:31:57.351819 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.351824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.351909 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.352037 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.352925 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.354351 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.356227 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.357107 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.357267 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.361498 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.361519 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.361526 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.361581 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.365978 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.366966 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.366972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.366977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.367188 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.367733 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.367741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.367748 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.368179 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.368314 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.369166 (+ 852us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.369483 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":80}
I20260620 23:31:57.370126 14102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.353811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50586 (local address 127.0.96.252:38569)
0620 23:31:57.354429 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.354433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.354481 (+ 48us) server_negotiation.cc:407] Connection header received
0620 23:31:57.354529 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.354534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.354613 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.355601 (+ 988us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.355839 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.356849 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.358449 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.359100 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.359266 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.363174 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.363179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.363182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.363210 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.367217 (+ 4007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.368022 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.368026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.368029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.368167 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.368518 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.368521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.368523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.368731 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.368866 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.369484 (+ 618us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.369681 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":317,"threads_started":1}
I20260620 23:31:57.370637 14100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.353567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50586)
0620 23:31:57.353731 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.353750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:57.353842 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:57.354759 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:57.354763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:57.355255 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:57.355457 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.355468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.356977 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.357011 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.358268 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.358286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.359324 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.359334 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.359608 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.360240 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:57.360267 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:57.362917 (+ 2650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:57.367389 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.367393 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.367400 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.367886 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.368281 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.368285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.368288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.368424 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.368853 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:57.368857 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:57.369063 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 23:31:57.369209 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 23:31:57.370301 (+ 1092us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":767}
I20260620 23:31:57.373744 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.351732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.0.96.253:40383)
0620 23:31:57.351869 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.351874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.352351 (+ 477us) server_negotiation.cc:407] Connection header received
0620 23:31:57.352431 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.352435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.352542 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.352667 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.354827 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.355631 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.357584 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.358331 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.358507 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.365508 (+ 7001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.365518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.365527 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.365601 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.369555 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.370843 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.370849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.370851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.371032 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.371507 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.371512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.371516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.371926 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.372261 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.373050 (+ 789us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.373375 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 23:31:57.376771 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: Bootstrap starting.
I20260620 23:31:57.377218 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: Bootstrap starting.
I20260620 23:31:57.380376 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:31:57.381212 13946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:57.381330 14095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:57.382614 14095 log.cc:824] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:57.382658 13946 log.cc:824] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:57.386786 14095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: No bootstrap required, opened a new log
I20260620 23:31:57.393695 13946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: No bootstrap required, opened a new log
I20260620 23:31:57.403278 14095 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.405097 14095 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:57.405336 14095 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e6b5c733d94b27a1a9e385d204e06f, State: Initialized, Role: FOLLOWER
I20260620 23:31:57.406368 14095 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.409751 13946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.416473 13946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:57.418828 13946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2feea8db8b4842c1b1919b8cf85bf806, State: Initialized, Role: FOLLOWER
I20260620 23:31:57.419900 13946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.423542 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.424448 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:31:57.429730 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.399395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.0.96.252:38569)
0620 23:31:57.399650 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.399657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.408046 (+ 8389us) server_negotiation.cc:407] Connection header received
0620 23:31:57.408591 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.408598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.408687 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.408759 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.411300 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.412538 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.414979 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.415755 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.415864 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.420232 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.420240 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.420244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.420287 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.424883 (+ 4596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.426217 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.426224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.426229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.426390 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.427012 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.427024 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.427030 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.427695 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.428128 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.429045 (+ 917us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.429350 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:31:57.431257 14095 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:31:57.431296 13946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:31:57.432155 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.392917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.0.96.253:40383)
0620 23:31:57.395186 (+ 2269us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.395192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.405353 (+ 10161us) server_negotiation.cc:407] Connection header received
0620 23:31:57.405830 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.405837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.405961 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.406045 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.409878 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.411556 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.415463 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.416472 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.416623 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.422441 (+ 5818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.422449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.422454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.422504 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.428239 (+ 5735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.429374 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.429382 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.429386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.429579 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.429990 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.429995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.429998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.430397 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.430673 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.431316 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.431726 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2113}
I20260620 23:31:57.435691 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.393156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.0.96.254:34319)
0620 23:31:57.393363 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.393370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.405759 (+ 12389us) server_negotiation.cc:407] Connection header received
0620 23:31:57.410592 (+ 4833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.410598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.410688 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.410764 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.413287 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.414313 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.416749 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.417498 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.417599 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.424259 (+ 6660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.424268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.424275 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.424324 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.428232 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.433168 (+ 4936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.433174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.433177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.433344 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.433825 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.433833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.433837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.434264 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.434518 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.435051 (+ 533us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.435309 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:31:57.435809 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.436640 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 23:31:57.451210 14131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:31:57.451486 14131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 23:31:57.456146 14135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:31:57.456377 14135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:31:57.451571 13879 master_runner.cc:186] Error getting master registration for 127.0.96.253:40383: OK, instance_id { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" instance_seqno: 1781998316753122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:31:57.468539 13879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:31:57.458742 14027 master_runner.cc:186] Error getting master registration for 127.0.96.253:40383: OK, instance_id { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" instance_seqno: 1781998316753122 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:31:57.471191 14027 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:31:57.494781 14020 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } attempt: 5
Jun 20 23:31:57 dist-test-slave-v34r krb5kdc[13866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
I20260620 23:31:57.510207 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.495533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:38569 (local address 127.0.0.1:50612)
0620 23:31:57.495924 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 23:31:57.495947 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:31:57.496080 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:31:57.497312 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:31:57.497317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:31:57.497879 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:31:57.498153 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.498169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.499474 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.499482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.500397 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:31:57.500408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.501262 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.501268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:31:57.501470 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.502111 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:31:57.502129 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:31:57.504207 (+ 2078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:31:57.507549 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.507554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.507559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.507949 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.508360 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:31:57.508365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:31:57.508368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:31:57.508510 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:31:57.509132 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:31:57.509137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:31:57.509298 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 23:31:57.509460 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 23:31:57.509761 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260620 23:31:57.510236 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:31:57.496310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.0.96.252:38569)
0620 23:31:57.496730 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 23:31:57.496737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:31:57.496756 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:31:57.496824 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:31:57.496830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:31:57.496939 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:31:57.497127 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:31:57.498329 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.499332 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.500556 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:31:57.501113 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:31:57.501253 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:31:57.504401 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:31:57.504406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:31:57.504410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:31:57.504440 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:31:57.507383 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.508081 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.508088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.508092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.508233 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:31:57.508627 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:31:57.508632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:31:57.508637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:31:57.508957 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:31:57.509165 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:31:57.509564 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 23:31:57.509777 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310}
I20260620 23:31:57.516428 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604: Bootstrap starting.
W20260620 23:31:57.519091 14049 tablet.cc:2369] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:31:57.519855 14020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604: Neither blocks nor log segments found. Creating new log.
I20260620 23:31:57.520848 14020 log.cc:824] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604: Log is configured to *not* fsync() on all Append() calls
I20260620 23:31:57.523471 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604: No bootstrap required, opened a new log
I20260620 23:31:57.532730 14020 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.533208 14020 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:31:57.533337 14020 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bb0ef050144baeb09417754b9b0604, State: Initialized, Role: FOLLOWER
I20260620 23:31:57.533957 14020 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.534929 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.535403 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:31:57.536161 14020 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:31:57.565352 13953 master_runner.cc:263] No leader master found from master b9bb0ef050144baeb09417754b9b0604
I20260620 23:31:57.565553 13953 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 23:31:57.572310 14150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:31:57.572597 14150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:31:57.602185 14139 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:31:57.602515 14139 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.604195 14139 leader_election.cc:288] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2feea8db8b4842c1b1919b8cf85bf806 (127.0.96.254:34319), 62e6b5c733d94b27a1a9e385d204e06f (127.0.96.252:38569)
I20260620 23:31:57.605590 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9bb0ef050144baeb09417754b9b0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2feea8db8b4842c1b1919b8cf85bf806" is_pre_election: true
I20260620 23:31:57.606243 13921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9bb0ef050144baeb09417754b9b0604 in term 0.
I20260620 23:31:57.607012 13971 leader_election.cc:302] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2feea8db8b4842c1b1919b8cf85bf806, b9bb0ef050144baeb09417754b9b0604; no voters:
I20260620 23:31:57.608003 14069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9bb0ef050144baeb09417754b9b0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e6b5c733d94b27a1a9e385d204e06f" is_pre_election: true
I20260620 23:31:57.608613 14069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9bb0ef050144baeb09417754b9b0604 in term 0.
I20260620 23:31:57.609597 14139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:31:57.609750 14139 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:31:57.609845 14139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:57.613431 14139 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.614821 13921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9bb0ef050144baeb09417754b9b0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2feea8db8b4842c1b1919b8cf85bf806"
I20260620 23:31:57.615080 13921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:57.618971 13921 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9bb0ef050144baeb09417754b9b0604 in term 1.
I20260620 23:31:57.625180 13971 leader_election.cc:302] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2feea8db8b4842c1b1919b8cf85bf806, b9bb0ef050144baeb09417754b9b0604; no voters:
I20260620 23:31:57.629927 14069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b9bb0ef050144baeb09417754b9b0604" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62e6b5c733d94b27a1a9e385d204e06f"
I20260620 23:31:57.630180 14069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:31:57.634377 14069 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9bb0ef050144baeb09417754b9b0604 in term 1.
I20260620 23:31:57.635294 14139 leader_election.cc:288] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [CANDIDATE]: Term 1 election: Requested vote from peers 2feea8db8b4842c1b1919b8cf85bf806 (127.0.96.254:34319), 62e6b5c733d94b27a1a9e385d204e06f (127.0.96.252:38569)
I20260620 23:31:57.635847 14139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:31:57.636354 14139 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [term 1 LEADER]: Becoming Leader. State: Replica: b9bb0ef050144baeb09417754b9b0604, State: Running, Role: LEADER
I20260620 23:31:57.638180 14139 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } }
I20260620 23:31:57.657397 14152 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bb0ef050144baeb09417754b9b0604. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.658007 14152 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:57.665036 14154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:31:57.667604 14154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260620 23:31:57.674197 13900 tablet.cc:2369] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:31:57.691344 14069 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [term 1 FOLLOWER]: Refusing update from remote peer b9bb0ef050144baeb09417754b9b0604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:31:57.692612 14152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:31:57.708392 13921 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [term 1 FOLLOWER]: Refusing update from remote peer b9bb0ef050144baeb09417754b9b0604: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:31:57.709731 14152 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:31:57.729352 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bb0ef050144baeb09417754b9b0604. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.729749 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:31:57.747720 14155 mvcc.cc:205] Tried to move back new op lower bound from 7299065109239861248 to 7299065109078437888. Current Snapshot: MvccSnapshot[applied={T|T < 7299065109239861248}]
I20260620 23:31:57.758714 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bb0ef050144baeb09417754b9b0604. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.759126 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:31:57.763682 14157 mvcc.cc:205] Tried to move back new op lower bound from 7299065109239861248 to 7299065109078437888. Current Snapshot: MvccSnapshot[applied={T|T < 7299065109239861248}]
I20260620 23:31:57.768909 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.769304 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:57.769728 14139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.770004 14139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604 [sys.catalog]: This master's current role is: LEADER
I20260620 23:31:57.774571 14154 catalog_manager.cc:1347] Generated new cluster ID: 09c41a15cc614decb8aba5e098d4fc89
I20260620 23:31:57.774700 14154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:31:57.776150 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.776472 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:31:57.787065 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9bb0ef050144baeb09417754b9b0604" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2feea8db8b4842c1b1919b8cf85bf806" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34319 } } peers { permanent_uuid: "b9bb0ef050144baeb09417754b9b0604" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 40383 } } peers { permanent_uuid: "62e6b5c733d94b27a1a9e385d204e06f" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 38569 } } }
I20260620 23:31:57.787364 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:31:57.830909 14154 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:31:57.837733 14154 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:31:57.867488 14154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9bb0ef050144baeb09417754b9b0604: Generated new TSK 0
I20260620 23:31:57.868788 14154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:31:58.454311 14131 catalog_manager.cc:1259] Loaded cluster ID: 09c41a15cc614decb8aba5e098d4fc89
I20260620 23:31:58.454465 14131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: loading cluster ID for follower catalog manager: success
I20260620 23:31:58.458120 14131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: acquiring CA information for follower catalog manager: success
I20260620 23:31:58.459359 14135 catalog_manager.cc:1259] Loaded cluster ID: 09c41a15cc614decb8aba5e098d4fc89
I20260620 23:31:58.459507 14135 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: loading cluster ID for follower catalog manager: success
I20260620 23:31:58.461458 14131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62e6b5c733d94b27a1a9e385d204e06f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:31:58.462925 14135 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: acquiring CA information for follower catalog manager: success
I20260620 23:31:58.465489 14135 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2feea8db8b4842c1b1919b8cf85bf806: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:31:58.501796 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13879
I20260620 23:31:58.503620 13941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:58.620193 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13879
I20260620 23:31:58.643759 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 13953
I20260620 23:31:58.645238 14015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:58.771762 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 13953
W20260620 23:31:58.793017 14046 connection.cc:535] server connection from 127.0.0.1:50612 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:31:58.793445 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14027
I20260620 23:31:58.794829 14090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:31:58.906915 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14027
2026-06-20T23:31:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4603 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:31:59 dist-test-slave-v34r krb5kdc[14171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:31:59 dist-test-slave-v34r krb5kdc[14171](info): set up 2 sockets
Jun 20 23:31:59 dist-test-slave-v34r krb5kdc[14171](info): commencing operation
krb5kdc: starting...
W20260620 23:32:00.034236 387 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:32:00 dist-test-slave-v34r krb5kdc[14171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:32:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:32:00.326257 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42085
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:32847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:00.694425 14184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:00.694746 14184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:00.694870 14184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:00.702309 14184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:32:00.702435 14184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:00.702481 14184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:00.702595 14184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:32:00.702666 14184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:32:00.712808 14184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:32847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:00.714239 14184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:00.715566 14184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:00.728708 14189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:00.728821 14192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:00.729228 14190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:00.730742 14184 server_base.cc:1034] running on GCE node
I20260620 23:32:00.731573 14184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:00.733140 14184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:00.734414 14184 hybrid_clock.cc:648] HybridClock initialized: now 1781998320734343 us; error 86 us; skew 500 ppm
I20260620 23:32:00.734827 14184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:00 dist-test-slave-v34r krb5kdc[14171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:00.738704 14184 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:32:00.740676 14184 webserver.cc:460] Webserver started at http://127.0.96.254:33611/ using document root <none> and password file <none>
I20260620 23:32:00.741662 14184 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:00.741789 14184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:00.742117 14184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:00.745983 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "cecd1ae5019d4045ab8e203c5bf18a47"
format_stamp: "Formatted at 2026-06-20 23:32:00 on dist-test-slave-v34r"
server_key: "b4fac391bc687062884cc8c690b0da60"
server_key_iv: "8cf6489846d66cd695d6c09d38f96259"
server_key_version: "encryptionkey@0"
I20260620 23:32:00.746933 14184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "cecd1ae5019d4045ab8e203c5bf18a47"
format_stamp: "Formatted at 2026-06-20 23:32:00 on dist-test-slave-v34r"
server_key: "b4fac391bc687062884cc8c690b0da60"
server_key_iv: "8cf6489846d66cd695d6c09d38f96259"
server_key_version: "encryptionkey@0"
I20260620 23:32:00.753835 14184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:32:00.758802 14199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:00.760833 14184 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:32:00.761220 14184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "cecd1ae5019d4045ab8e203c5bf18a47"
format_stamp: "Formatted at 2026-06-20 23:32:00 on dist-test-slave-v34r"
server_key: "b4fac391bc687062884cc8c690b0da60"
server_key_iv: "8cf6489846d66cd695d6c09d38f96259"
server_key_version: "encryptionkey@0"
I20260620 23:32:00.761541 14184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:00.783725 14184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:00.789690 14184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:00.790079 14184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:00.815749 14184 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:42085
I20260620 23:32:00.815789 14250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:42085 every 8 connection(s)
I20260620 23:32:00.817715 14184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:32:00.822248 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:00.825917 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14184
I20260620 23:32:00.826215 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:32:00.826989 387 external_mini_cluster.cc:1428] Setting key 9ed0e9bb96425a48a266e2ecba9af04a
I20260620 23:32:00.833590 14251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47: Bootstrap starting.
I20260620 23:32:00.837731 14251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:00.839087 14251 log.cc:824] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:00.843381 14251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47: No bootstrap required, opened a new log
Jun 20 23:32:00 dist-test-slave-v34r krb5kdc[14171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:00.852119 14251 raft_consensus.cc:348] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42085 } }
I20260620 23:32:00.852502 14251 raft_consensus.cc:374] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:00.852626 14251 raft_consensus.cc:729] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cecd1ae5019d4045ab8e203c5bf18a47, State: Initialized, Role: FOLLOWER
I20260620 23:32:00.853404 14251 consensus_queue.cc:260] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42085 } }
I20260620 23:32:00.853710 14251 raft_consensus.cc:388] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:32:00.853860 14251 raft_consensus.cc:482] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:32:00.854072 14251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:00.856637 14251 raft_consensus.cc:504] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42085 } }
I20260620 23:32:00.857354 14251 leader_election.cc:302] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cecd1ae5019d4045ab8e203c5bf18a47; no voters:
I20260620 23:32:00.858098 14251 leader_election.cc:288] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:32:00.858227 14256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:00.858666 14254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:00.829423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.0.96.254:42085)
0620 23:32:00.830554 (+ 1131us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:00.830570 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:00.830632 (+ 62us) server_negotiation.cc:407] Connection header received
0620 23:32:00.832786 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:00.832821 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:00.833488 (+ 667us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:00.834006 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:00.836060 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:00.837900 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:00.840229 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:00.841496 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:00.841702 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:00.847391 (+ 5689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:00.847404 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:00.847432 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:00.847497 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:00.851657 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:00.853219 (+ 1562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:00.853232 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:00.853242 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:00.853559 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:00.854301 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:00.854306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:00.854308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:00.855184 (+ 876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:00.855622 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:00.856183 (+ 561us) server_negotiation.cc:299] Negotiation successful
0620 23:32:00.856810 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":508,"threads_started":1}
I20260620 23:32:00.859642 14256 raft_consensus.cc:686] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [term 1 LEADER]: Becoming Leader. State: Replica: cecd1ae5019d4045ab8e203c5bf18a47, State: Running, Role: LEADER
I20260620 23:32:00.860407 14256 consensus_queue.cc:237] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42085 } }
I20260620 23:32:00.860857 14251 sys_catalog.cc:564] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:32:00.864639 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:32:00.865242 14257 sys_catalog.cc:455] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42085 } } }
I20260620 23:32:00.865319 14258 sys_catalog.cc:455] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cecd1ae5019d4045ab8e203c5bf18a47. Latest consensus state: current_term: 1 leader_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cecd1ae5019d4045ab8e203c5bf18a47" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42085 } } }
I20260620 23:32:00.865779 14257 sys_catalog.cc:458] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:00.865825 14258 sys_catalog.cc:458] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47 [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:00.873338 14265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:32:00.879462 14265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:32:00.890262 14265 catalog_manager.cc:1347] Generated new cluster ID: c27cf98c497a4257890393590b1afd09
I20260620 23:32:00.890434 14265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:32:00.943639 14265 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:32:00.945246 14265 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:32:00.952870 14265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cecd1ae5019d4045ab8e203c5bf18a47: Generated new TSK 0
I20260620 23:32:00.953861 14265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:32:00.982896 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14184
I20260620 23:32:00.984864 14246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:01.086839 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14184
2026-06-20T23:32:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2177 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:32:01 dist-test-slave-v34r krb5kdc[14279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:32:01 dist-test-slave-v34r krb5kdc[14279](info): set up 2 sockets
Jun 20 23:32:01 dist-test-slave-v34r krb5kdc[14279](info): commencing operation
krb5kdc: starting...
W20260620 23:32:02.225553 387 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:32:02 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:32:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:32:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:32:02.501186 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35269
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:02.848799 14292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:02.849117 14292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:02.849210 14292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:02.856348 14292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:32:02.856473 14292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:02.856520 14292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:02.856578 14292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:32:02.856617 14292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:32:02.866581 14292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:02.867902 14292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:02.869216 14292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:02.883069 14297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:02.882936 14298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:02.882870 14300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:02.884446 14292 server_base.cc:1034] running on GCE node
I20260620 23:32:02.885452 14292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:02.887102 14292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:02.888339 14292 hybrid_clock.cc:648] HybridClock initialized: now 1781998322888295 us; error 44 us; skew 500 ppm
I20260620 23:32:02.888751 14292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:02 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:02.892323 14292 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:32:02.894197 14292 webserver.cc:460] Webserver started at http://127.0.96.254:35437/ using document root <none> and password file <none>
I20260620 23:32:02.894972 14292 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:02.895135 14292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:02.895462 14292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:02.898711 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "f7225466d04a4092afb14e25a3fbd8cb"
format_stamp: "Formatted at 2026-06-20 23:32:02 on dist-test-slave-v34r"
server_key: "4643cca18a9f09ff6e10311eca76469b"
server_key_iv: "399ce63d395b7dc7153719f844d1b82d"
server_key_version: "encryptionkey@0"
I20260620 23:32:02.899544 14292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "f7225466d04a4092afb14e25a3fbd8cb"
format_stamp: "Formatted at 2026-06-20 23:32:02 on dist-test-slave-v34r"
server_key: "4643cca18a9f09ff6e10311eca76469b"
server_key_iv: "399ce63d395b7dc7153719f844d1b82d"
server_key_version: "encryptionkey@0"
I20260620 23:32:02.905650 14292 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:32:02.910290 14307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:02.912492 14292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:32:02.912734 14292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "f7225466d04a4092afb14e25a3fbd8cb"
format_stamp: "Formatted at 2026-06-20 23:32:02 on dist-test-slave-v34r"
server_key: "4643cca18a9f09ff6e10311eca76469b"
server_key_iv: "399ce63d395b7dc7153719f844d1b82d"
server_key_version: "encryptionkey@0"
I20260620 23:32:02.913106 14292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:02.930262 14292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:02.935204 14292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:02.935523 14292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:02.957587 14292 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:35269
I20260620 23:32:02.957652 14358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:35269 every 8 connection(s)
I20260620 23:32:02.959326 14292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:32:02.960707 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14292
I20260620 23:32:02.961023 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:32:02.961951 387 external_mini_cluster.cc:1428] Setting key 6c69e68ba0b523d5443a1b34e05c6cb1
I20260620 23:32:02.965926 14359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:32:02 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:02.984334 14359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap starting.
I20260620 23:32:02.990002 14359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:02.991852 14359 log.cc:824] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:02.994187 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:02.964645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.0.96.254:35269)
0620 23:32:02.965868 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:02.965882 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:02.965939 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:32:02.967865 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:02.967891 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:02.968385 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:02.968802 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:02.971148 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:02.973097 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:02.975641 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:02.976582 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:02.976782 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:02.982823 (+ 6041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:02.982839 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:02.982870 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:02.982944 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:02.987705 (+ 4761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:02.988755 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:02.988768 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:02.988779 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:02.989118 (+ 339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:02.989660 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:02.989666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:02.989669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:02.990692 (+ 1023us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:02.991067 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:02.991625 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 23:32:02.992238 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":476,"threads_started":1}
I20260620 23:32:02.996747 14359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: No bootstrap required, opened a new log
I20260620 23:32:03.007046 14359 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:03.007735 14359 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:03.007871 14359 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7225466d04a4092afb14e25a3fbd8cb, State: Initialized, Role: FOLLOWER
I20260620 23:32:03.008739 14359 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:03.009097 14359 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:32:03.009255 14359 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:32:03.009469 14359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:03.012465 14359 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:03.013312 14359 leader_election.cc:302] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7225466d04a4092afb14e25a3fbd8cb; no voters:
I20260620 23:32:03.014204 14359 leader_election.cc:288] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:32:03.014498 14364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:03.015656 14364 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 LEADER]: Becoming Leader. State: Replica: f7225466d04a4092afb14e25a3fbd8cb, State: Running, Role: LEADER
I20260620 23:32:03.016467 14364 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:03.017519 14359 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:32:03.020645 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7225466d04a4092afb14e25a3fbd8cb. Latest consensus state: current_term: 1 leader_uuid: "f7225466d04a4092afb14e25a3fbd8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } } }
I20260620 23:32:03.021616 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:03.020725 14365 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7225466d04a4092afb14e25a3fbd8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } } }
I20260620 23:32:03.022831 14365 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:03.022918 14373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:32:03.033218 14373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:32:03.046100 14373 catalog_manager.cc:1347] Generated new cluster ID: 42f06effb09a4b959a34f9378f3bec0a
I20260620 23:32:03.046252 14373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:32:03.078951 14373 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:32:03.080837 14373 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:32:03.089414 14373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Generated new TSK 0
I20260620 23:32:03.090651 14373 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:32:03.163230 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:35269
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:03.529906 14385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:03.530201 14385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:03.530296 14385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:03.537595 14385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:03.537721 14385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:03.537835 14385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:32:03.547518 14385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:03.548748 14385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:03.550019 14385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:03.563705 14393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:03.563886 14390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:03.563835 14391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:03.565209 14385 server_base.cc:1034] running on GCE node
I20260620 23:32:03.565987 14385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:03.567214 14385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:03.568470 14385 hybrid_clock.cc:648] HybridClock initialized: now 1781998323568387 us; error 45 us; skew 500 ppm
I20260620 23:32:03.568918 14385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:03 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:03.572697 14385 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:32:03.574693 14385 webserver.cc:460] Webserver started at http://127.0.96.193:39049/ using document root <none> and password file <none>
I20260620 23:32:03.575592 14385 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:03.575760 14385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:03.576129 14385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:03.579967 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "d24a79d690ed4a25a24d9cef0e4edde6"
format_stamp: "Formatted at 2026-06-20 23:32:03 on dist-test-slave-v34r"
server_key: "6a64dd5abe9f0a3e1736f1f5908f846e"
server_key_iv: "a50f6e867b46f429858f12ad65e5a666"
server_key_version: "encryptionkey@0"
I20260620 23:32:03.581085 14385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "d24a79d690ed4a25a24d9cef0e4edde6"
format_stamp: "Formatted at 2026-06-20 23:32:03 on dist-test-slave-v34r"
server_key: "6a64dd5abe9f0a3e1736f1f5908f846e"
server_key_iv: "a50f6e867b46f429858f12ad65e5a666"
server_key_version: "encryptionkey@0"
I20260620 23:32:03.588215 14385 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 23:32:03.593472 14400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:03.595544 14385 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:32:03.595815 14385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "d24a79d690ed4a25a24d9cef0e4edde6"
format_stamp: "Formatted at 2026-06-20 23:32:03 on dist-test-slave-v34r"
server_key: "6a64dd5abe9f0a3e1736f1f5908f846e"
server_key_iv: "a50f6e867b46f429858f12ad65e5a666"
server_key_version: "encryptionkey@0"
I20260620 23:32:03.596091 14385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:03.636451 14385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:03.642225 14385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:03.642625 14385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:03.643926 14385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:03.645843 14385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:03.645931 14385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:03.646134 14385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:03.646201 14385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:03.687685 14385 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:42903
I20260620 23:32:03.687801 14512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:42903 every 8 connection(s)
I20260620 23:32:03.689494 14385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:32:03.697059 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14385
I20260620 23:32:03.697319 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:32:03.698184 387 external_mini_cluster.cc:1428] Setting key 404ef77094b520143d1cdbdfbaa5ae44
Jun 20 23:32:03 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:03.718397 14515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:03.693116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:43867 (local address 127.0.96.254:35269)
0620 23:32:03.693772 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:03.693780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:03.695603 (+ 1823us) server_negotiation.cc:407] Connection header received
0620 23:32:03.698545 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:03.698552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:03.698658 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:03.698803 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:03.700576 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:03.701577 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:03.704388 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:03.704840 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:03.704966 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:03.710018 (+ 5052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:03.710028 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:03.710035 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:03.710097 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:03.714339 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:03.715227 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:03.715236 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:03.715243 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:03.715441 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:03.715898 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:03.715906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:03.715912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:03.716350 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:03.716552 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:03.717454 (+ 902us) server_negotiation.cc:299] Negotiation successful
0620 23:32:03.717801 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
I20260620 23:32:03.719463 14516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:03.693980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:43867)
0620 23:32:03.695361 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:03.695430 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:03.698007 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:03.699100 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:03.699109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:03.699699 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:03.700371 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:03.700390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:03.701759 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:03.701768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:03.704158 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:03.704177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:03.705159 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:03.705169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:03.705525 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:03.706459 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:03.706501 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:03.709651 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:03.714551 (+ 4900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:03.714560 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:03.714579 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:03.715065 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:03.715579 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:03.715582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:03.715585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:03.715768 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:03.716488 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:03.716494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:03.716908 (+ 414us) client_negotiation.cc:769] Sending connection context
0620 23:32:03.717272 (+ 364us) client_negotiation.cc:240] Negotiation successful
0620 23:32:03.717878 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":526,"threads_started":1}
I20260620 23:32:03.721272 14513 heartbeater.cc:347] Connected to a master server at 127.0.96.254:35269
I20260620 23:32:03.721630 14513 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:03.722388 14513 heartbeater.cc:511] Master 127.0.96.254:35269 requested a full tablet report, sending...
I20260620 23:32:03.724903 14324 ts_manager.cc:194] Registered new tserver with Master: d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193:42903)
I20260620 23:32:03.726538 14324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:43867
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:32:03.761709 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:35269
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:04.185650 14519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:04.186038 14519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:04.186152 14519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:04.194866 14519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:04.195047 14519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:04.195187 14519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:32:04.206890 14519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:04.208612 14519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:04.210376 14519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:04.228710 14525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:04.228822 14527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:04.228796 14524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:04.230144 14519 server_base.cc:1034] running on GCE node
I20260620 23:32:04.231025 14519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:04.232321 14519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:04.233593 14519 hybrid_clock.cc:648] HybridClock initialized: now 1781998324233519 us; error 50 us; skew 500 ppm
I20260620 23:32:04.234047 14519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:04 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:04.238304 14519 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:32:04.240525 14519 webserver.cc:460] Webserver started at http://127.0.96.194:35695/ using document root <none> and password file <none>
I20260620 23:32:04.241602 14519 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:04.241787 14519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:04.242180 14519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:04.246049 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "e26660f20cae49019671ba389347835f"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "ef6e8870c50384ffbd14218606ea70e7"
server_key_iv: "fd33b3854881e4e31196f3880b84417a"
server_key_version: "encryptionkey@0"
I20260620 23:32:04.247088 14519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "e26660f20cae49019671ba389347835f"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "ef6e8870c50384ffbd14218606ea70e7"
server_key_iv: "fd33b3854881e4e31196f3880b84417a"
server_key_version: "encryptionkey@0"
I20260620 23:32:04.254997 14519 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20260620 23:32:04.260346 14534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:04.262447 14519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:32:04.262741 14519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "e26660f20cae49019671ba389347835f"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "ef6e8870c50384ffbd14218606ea70e7"
server_key_iv: "fd33b3854881e4e31196f3880b84417a"
server_key_version: "encryptionkey@0"
I20260620 23:32:04.263046 14519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:04.292272 14519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:04.298655 14519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:04.299082 14519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:04.300419 14519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:04.302402 14519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:04.302531 14519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:04.302707 14519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:04.302793 14519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:04.346458 14519 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34257
I20260620 23:32:04.346494 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34257 every 8 connection(s)
I20260620 23:32:04.348344 14519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:32:04.352355 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14519
I20260620 23:32:04.352587 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:32:04.353271 387 external_mini_cluster.cc:1428] Setting key c544a25aef29aed5973e0bac2cc05acd
Jun 20 23:32:04 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:04.376969 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.352001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:55667 (local address 127.0.96.254:35269)
0620 23:32:04.352554 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:04.352560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:04.354892 (+ 2332us) server_negotiation.cc:407] Connection header received
0620 23:32:04.357090 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:04.357100 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:04.357230 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:04.357397 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:04.359202 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.360520 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.362538 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.363424 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.363621 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:04.368515 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:04.368521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:04.368525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:04.368575 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:04.372209 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:04.373873 (+ 1664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:04.373879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:04.373882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:04.374058 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:04.374619 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:04.374631 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:04.374634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:04.375005 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:04.375191 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:04.376202 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0620 23:32:04.376529 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
I20260620 23:32:04.378322 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.352956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:55667)
0620 23:32:04.354286 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:04.354373 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:04.356687 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:04.357667 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:04.357675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:04.358294 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:04.358944 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:04.358961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.360749 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.360753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:04.362223 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:04.362237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.363719 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.363724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:04.363954 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:04.364734 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:04.364760 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:04.368100 (+ 3340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:04.372500 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:04.372514 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:04.372543 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:04.373608 (+ 1065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:04.374199 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:04.374205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:04.374209 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:04.374476 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:04.375157 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:04.375167 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:04.375725 (+ 558us) client_negotiation.cc:769] Sending connection context
0620 23:32:04.376078 (+ 353us) client_negotiation.cc:240] Negotiation successful
0620 23:32:04.376814 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":436,"threads_started":1}
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:32:04.380594 14647 heartbeater.cc:347] Connected to a master server at 127.0.96.254:35269
I20260620 23:32:04.381122 14647 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:04.382138 14647 heartbeater.cc:511] Master 127.0.96.254:35269 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:32:04.384501 14324 ts_manager.cc:194] Registered new tserver with Master: e26660f20cae49019671ba389347835f (127.0.96.194:34257)
I20260620 23:32:04.385594 14324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:55667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:32:04.415078 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:35269
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:32:04.730448 14513 heartbeater.cc:503] Master 127.0.96.254:35269 was elected leader, sending a full tablet report...
W20260620 23:32:04.771901 14653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:04.772198 14653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:04.772295 14653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:04.779654 14653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:04.779788 14653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:04.779904 14653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:32:04.789898 14653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:04.791201 14653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:04.792616 14653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:04.807247 14659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:04.807338 14658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:04.807358 14661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:04.809082 14653 server_base.cc:1034] running on GCE node
I20260620 23:32:04.809805 14653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:04.810901 14653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:04.812162 14653 hybrid_clock.cc:648] HybridClock initialized: now 1781998324812111 us; error 59 us; skew 500 ppm
I20260620 23:32:04.812516 14653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:04 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:04.816051 14653 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:32:04.817986 14653 webserver.cc:460] Webserver started at http://127.0.96.195:37987/ using document root <none> and password file <none>
I20260620 23:32:04.818753 14653 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:04.818909 14653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:04.819236 14653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:04.822546 14653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "965f5613608db0cab6e8b682e3d871c3"
server_key_iv: "12e3441a4664b41a2a0b77b8d0d47778"
server_key_version: "encryptionkey@0"
I20260620 23:32:04.823438 14653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "965f5613608db0cab6e8b682e3d871c3"
server_key_iv: "12e3441a4664b41a2a0b77b8d0d47778"
server_key_version: "encryptionkey@0"
I20260620 23:32:04.829618 14653 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 23:32:04.833866 14668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:04.835538 14653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:32:04.835772 14653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "965f5613608db0cab6e8b682e3d871c3"
server_key_iv: "12e3441a4664b41a2a0b77b8d0d47778"
server_key_version: "encryptionkey@0"
I20260620 23:32:04.835997 14653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:04.856194 14653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:04.860729 14653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:04.861080 14653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:04.862257 14653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:04.863852 14653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:04.863950 14653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:04.864107 14653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:04.864180 14653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:04.904834 14653 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:38007
I20260620 23:32:04.905056 14780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:38007 every 8 connection(s)
I20260620 23:32:04.906270 14653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:32:04.915314 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14653
I20260620 23:32:04.915475 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:32:04.916139 387 external_mini_cluster.cc:1428] Setting key bc757c394aa79ae09cc29ca8c9f25be9
Jun 20 23:32:04 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998324, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:04.933466 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.909578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:42701 (local address 127.0.96.254:35269)
0620 23:32:04.910130 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:04.910135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:04.911333 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 23:32:04.913625 (+ 2292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:04.913632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:04.913768 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:04.913910 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:04.916156 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.917328 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.919709 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.920529 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.920689 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:04.925791 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:04.925799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:04.925803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:04.925867 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:04.929096 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:04.930106 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:04.930114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:04.930118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:04.930301 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:04.930977 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:04.930986 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:04.930992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:04.931528 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:04.931755 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:04.932761 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0620 23:32:04.933119 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":254,"threads_started":1}
I20260620 23:32:04.934330 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.910126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:42701)
0620 23:32:04.911074 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:04.911143 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:04.913254 (+ 2111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:04.914215 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:04.914226 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:04.914930 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:04.915877 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:04.915912 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.917525 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.917531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:04.919532 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:04.919551 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.920826 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.920835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:04.921224 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:04.922199 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:04.922239 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:04.925461 (+ 3222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:04.929344 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:04.929352 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:04.929368 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:04.929930 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:04.930445 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:04.930450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:04.930459 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:04.930717 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:04.931716 (+ 999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:04.931732 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:04.932195 (+ 463us) client_negotiation.cc:769] Sending connection context
0620 23:32:04.932561 (+ 366us) client_negotiation.cc:240] Negotiation successful
0620 23:32:04.933136 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":364,"threads_started":1}
I20260620 23:32:04.936299 14781 heartbeater.cc:347] Connected to a master server at 127.0.96.254:35269
I20260620 23:32:04.936792 14781 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:04.937868 14781 heartbeater.cc:511] Master 127.0.96.254:35269 requested a full tablet report, sending...
I20260620 23:32:04.940084 14324 ts_manager.cc:194] Registered new tserver with Master: 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
I20260620 23:32:04.941252 14324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:42701
I20260620 23:32:04.945683 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:32:04.967978 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.951299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.96.254:35269)
0620 23:32:04.951656 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:04.951665 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:04.952248 (+ 583us) server_negotiation.cc:407] Connection header received
0620 23:32:04.952609 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:04.952615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:04.952720 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:04.952858 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:04.954721 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.955580 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.957277 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:04.957775 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:04.957939 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:04.961129 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:04.961140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:04.961153 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:04.961221 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:04.964651 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:04.965661 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:04.965667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:04.965672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:04.965821 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:04.966220 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:04.966227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:04.966235 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:04.966644 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:04.966946 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:04.967280 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 23:32:04.967584 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 23:32:04.973733 14324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:32:04.982193 14324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:32:05.011840 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.999123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:38007 (local address 127.0.0.1:41954)
0620 23:32:05.000535 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:05.000597 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:05.000833 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:05.002584 (+ 1751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:05.002597 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:05.002642 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:05.003399 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.003422 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.006040 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.006046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.008389 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.008405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.010653 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.010660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.010949 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.011028 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 23:32:05.011180 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 23:32:05.011422 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"thread_start_us":2393,"threads_started":1}
I20260620 23:32:05.013404 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.999611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.0.96.195:38007)
0620 23:32:05.001658 (+ 2047us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.001669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.001711 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:32:05.001842 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.001854 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.002217 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.003703 (+ 1486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:05.004068 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.005834 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.008625 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.010372 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.012294 (+ 1922us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.012546 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.012650 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.012892 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1829,"thread_start_us":1623,"threads_started":1}
I20260620 23:32:05.013830 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:04.999590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34257 (local address 127.0.0.1:45646)
0620 23:32:05.001862 (+ 2272us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:05.001889 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:05.002698 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:05.005272 (+ 2574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:05.005277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:05.005305 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:05.005799 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.005813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.007894 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.007902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.009957 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.009975 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.012679 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.012685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.012900 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.012940 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:32:05.013104 (+ 164us) client_negotiation.cc:240] Negotiation successful
0620 23:32:05.013314 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2115,"mutex_wait_us":17,"thread_start_us":428,"threads_started":1}
I20260620 23:32:05.014228 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.002433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.0.96.193:42903)
0620 23:32:05.005181 (+ 2748us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.005191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.005224 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:32:05.005332 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.005341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.005619 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.005881 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:05.006978 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.008293 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.011109 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.012800 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.012966 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.013247 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.013660 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.013844 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2492,"thread_start_us":2263,"threads_started":1}
I20260620 23:32:05.014881 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.002057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:44440)
0620 23:32:05.003580 (+ 1523us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:05.003612 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:05.003749 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:05.006210 (+ 2461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:05.006215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:05.006251 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:05.006735 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.006748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.008461 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.008469 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.010889 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.010906 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.013092 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.013099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.013302 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.013339 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 23:32:05.013437 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 23:32:05.014538 (+ 1101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"thread_start_us":1041,"threads_started":1}
I20260620 23:32:05.015311 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.001088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.0.96.194:34257)
0620 23:32:05.004413 (+ 3325us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.004429 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.004474 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:32:05.004643 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.004657 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.004975 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.006048 (+ 1073us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:05.006333 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.007712 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.010203 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.011631 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.012226 (+ 595us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.012500 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.013401 (+ 901us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.014941 (+ 1540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3054,"thread_start_us":3104,"threads_started":1}
I20260620 23:32:05.019555 14582 tablet_service.cc:1467] Processing CreateTablet for tablet 86397d343a29436eab33387e398fb951 (DEFAULT_TABLE table=test-table [id=6448a61bbbcc44bbbd4d5aa9b9e8a252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:05.019423 14716 tablet_service.cc:1467] Processing CreateTablet for tablet 86397d343a29436eab33387e398fb951 (DEFAULT_TABLE table=test-table [id=6448a61bbbcc44bbbd4d5aa9b9e8a252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:05.019902 14448 tablet_service.cc:1467] Processing CreateTablet for tablet 86397d343a29436eab33387e398fb951 (DEFAULT_TABLE table=test-table [id=6448a61bbbcc44bbbd4d5aa9b9e8a252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:05.020840 14582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86397d343a29436eab33387e398fb951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:05.021533 14716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86397d343a29436eab33387e398fb951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:05.022331 14448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86397d343a29436eab33387e398fb951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:05.038107 14800 tablet_bootstrap.cc:492] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Bootstrap starting.
I20260620 23:32:05.039599 14801 tablet_bootstrap.cc:492] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Bootstrap starting.
I20260620 23:32:05.042976 14802 tablet_bootstrap.cc:492] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Bootstrap starting.
I20260620 23:32:05.043982 14800 tablet_bootstrap.cc:654] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:05.045499 14801 tablet_bootstrap.cc:654] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:05.046798 14801 log.cc:824] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:05.046926 14800 log.cc:824] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:05.047967 14802 tablet_bootstrap.cc:654] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:05.049943 14802 log.cc:824] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:05.050534 14800 tablet_bootstrap.cc:492] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: No bootstrap required, opened a new log
I20260620 23:32:05.050875 14800 ts_tablet_manager.cc:1397] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260620 23:32:05.051261 14801 tablet_bootstrap.cc:492] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: No bootstrap required, opened a new log
I20260620 23:32:05.051721 14801 ts_tablet_manager.cc:1397] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260620 23:32:05.057406 14802 tablet_bootstrap.cc:492] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: No bootstrap required, opened a new log
I20260620 23:32:05.057958 14802 ts_tablet_manager.cc:1397] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260620 23:32:05.059837 14800 raft_consensus.cc:348] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.060279 14800 raft_consensus.cc:374] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:05.060372 14800 raft_consensus.cc:729] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eee1b37e4de40bb9c43e2eb26762cb9, State: Initialized, Role: FOLLOWER
I20260620 23:32:05.061146 14800 consensus_queue.cc:260] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.062404 14781 heartbeater.cc:503] Master 127.0.96.254:35269 was elected leader, sending a full tablet report...
I20260620 23:32:05.063088 14800 ts_tablet_manager.cc:1428] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260620 23:32:05.064287 14801 raft_consensus.cc:348] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.064800 14801 raft_consensus.cc:374] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:05.064905 14801 raft_consensus.cc:729] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26660f20cae49019671ba389347835f, State: Initialized, Role: FOLLOWER
I20260620 23:32:05.065901 14801 consensus_queue.cc:260] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.073778 14647 heartbeater.cc:503] Master 127.0.96.254:35269 was elected leader, sending a full tablet report...
I20260620 23:32:05.073163 14802 raft_consensus.cc:348] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.074051 14802 raft_consensus.cc:374] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:05.074223 14802 raft_consensus.cc:729] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24a79d690ed4a25a24d9cef0e4edde6, State: Initialized, Role: FOLLOWER
I20260620 23:32:05.075162 14802 consensus_queue.cc:260] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.075551 14801 ts_tablet_manager.cc:1428] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
I20260620 23:32:05.077469 14802 ts_tablet_manager.cc:1428] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Time spent starting tablet: real 0.019s user 0.019s sys 0.001s
W20260620 23:32:05.101581 14648 tablet.cc:2369] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:05.152910 14808 raft_consensus.cc:482] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:05.153314 14808 raft_consensus.cc:504] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.155547 14808 leader_election.cc:288] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007), e26660f20cae49019671ba389347835f (127.0.96.194:34257)
W20260620 23:32:05.158141 14782 tablet.cc:2369] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:05.164615 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.155928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:55679 (local address 127.0.96.195:38007)
0620 23:32:05.156163 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.156172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.157013 (+ 841us) server_negotiation.cc:407] Connection header received
0620 23:32:05.157293 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.157299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.157393 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.157527 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:05.158454 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.159340 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.162000 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.163246 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.163429 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.163541 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.164172 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.164310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:32:05.164594 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.156266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:54063 (local address 127.0.96.194:34257)
0620 23:32:05.156551 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.156561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.157362 (+ 801us) server_negotiation.cc:407] Connection header received
0620 23:32:05.157548 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.157552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.157641 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.157743 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:05.158535 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.159587 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.162283 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.163468 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.163629 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.163678 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.164172 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.164310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:32:05.165017 14810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.156129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34257 (local address 127.0.96.193:54063)
0620 23:32:05.157213 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:05.157229 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:05.157389 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:05.157794 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:05.157797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:05.157816 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:05.158257 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.158269 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.159781 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.159786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.162110 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.162127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.163708 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.163714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.163966 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.163996 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:32:05.164123 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 23:32:05.164353 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":93,"thread_start_us":653,"threads_started":1}
I20260620 23:32:05.165046 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.155928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:38007 (local address 127.0.96.193:55679)
0620 23:32:05.156841 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:05.156867 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:05.157110 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:05.157590 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:05.157593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:05.157622 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:05.158191 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.158210 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.159514 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.159521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.161809 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:05.161823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.163579 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.163588 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:05.163935 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.163986 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:32:05.164120 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:32:05.164358 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":118,"thread_start_us":447,"threads_started":1}
I20260620 23:32:05.166124 14736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86397d343a29436eab33387e398fb951" candidate_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" is_pre_election: true
I20260620 23:32:05.166297 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86397d343a29436eab33387e398fb951" candidate_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26660f20cae49019671ba389347835f" is_pre_election: true
I20260620 23:32:05.166672 14736 raft_consensus.cc:2454] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24a79d690ed4a25a24d9cef0e4edde6 in term 0.
I20260620 23:32:05.167009 14602 raft_consensus.cc:2454] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d24a79d690ed4a25a24d9cef0e4edde6 in term 0.
I20260620 23:32:05.167421 14404 leader_election.cc:302] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eee1b37e4de40bb9c43e2eb26762cb9, d24a79d690ed4a25a24d9cef0e4edde6; no voters:
I20260620 23:32:05.167919 14808 raft_consensus.cc:2781] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:32:05.168097 14808 raft_consensus.cc:482] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:32:05.168218 14808 raft_consensus.cc:3037] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:05.171146 14808 raft_consensus.cc:504] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.172021 14808 leader_election.cc:288] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 1 election: Requested vote from peers 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007), e26660f20cae49019671ba389347835f (127.0.96.194:34257)
I20260620 23:32:05.172497 14736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86397d343a29436eab33387e398fb951" candidate_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
I20260620 23:32:05.172735 14736 raft_consensus.cc:3037] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:05.172731 14602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86397d343a29436eab33387e398fb951" candidate_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26660f20cae49019671ba389347835f"
I20260620 23:32:05.173074 14602 raft_consensus.cc:3037] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:05.175263 14736 raft_consensus.cc:2454] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24a79d690ed4a25a24d9cef0e4edde6 in term 1.
I20260620 23:32:05.175652 14602 raft_consensus.cc:2454] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d24a79d690ed4a25a24d9cef0e4edde6 in term 1.
I20260620 23:32:05.175782 14404 leader_election.cc:302] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eee1b37e4de40bb9c43e2eb26762cb9, d24a79d690ed4a25a24d9cef0e4edde6; no voters:
I20260620 23:32:05.176151 14808 raft_consensus.cc:2781] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:05.176685 14808 raft_consensus.cc:686] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 LEADER]: Becoming Leader. State: Replica: d24a79d690ed4a25a24d9cef0e4edde6, State: Running, Role: LEADER
I20260620 23:32:05.177320 14808 consensus_queue.cc:237] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } }
I20260620 23:32:05.182623 14323 catalog_manager.cc:5495] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193). New cstate: current_term: 1 leader_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } health_report { overall_health: HEALTHY } } }
W20260620 23:32:05.192468 14514 tablet.cc:2369] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:05.212422 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.206964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.0.96.193:42903)
0620 23:32:05.207235 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.207243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.207263 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:32:05.207407 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.207412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.207489 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.207663 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:32:05.208047 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.208754 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.210252 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.210650 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.210786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.211309 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:32:05.211809 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:32:05.211965 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.212044 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.212139 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:32:05.222891 14736 raft_consensus.cc:1261] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Refusing update from remote peer d24a79d690ed4a25a24d9cef0e4edde6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:05.223321 14602 raft_consensus.cc:1261] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Refusing update from remote peer d24a79d690ed4a25a24d9cef0e4edde6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:05.224046 14811 consensus_queue.cc:1035] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:05.224491 14808 consensus_queue.cc:1035] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:05.258598 14324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:05.259011 14324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:05.263736 14324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Sending DeleteTablet for 3 replicas of tablet 86397d343a29436eab33387e398fb951
I20260620 23:32:05.265467 14448 tablet_service.cc:1514] Processing DeleteTablet for tablet 86397d343a29436eab33387e398fb951 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:05 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:44440
I20260620 23:32:05.265555 14716 tablet_service.cc:1514] Processing DeleteTablet for tablet 86397d343a29436eab33387e398fb951 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:05 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:41954
I20260620 23:32:05.265537 14582 tablet_service.cc:1514] Processing DeleteTablet for tablet 86397d343a29436eab33387e398fb951 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:05 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:45646
I20260620 23:32:05.266036 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14385
I20260620 23:32:05.276705 14825 tablet_replica.cc:330] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: stopping tablet replica
I20260620 23:32:05.277091 14824 tablet_replica.cc:330] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: stopping tablet replica
I20260620 23:32:05.277415 14825 raft_consensus.cc:2229] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:05.277742 14824 raft_consensus.cc:2229] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:32:05.278160 14825 raft_consensus.cc:2258] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:05.278959 14824 raft_consensus.cc:2258] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:05.282192 14825 ts_tablet_manager.cc:1905] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:05.282701 14824 ts_tablet_manager.cc:1905] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:05.289526 14825 ts_tablet_manager.cc:1918] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:05.289714 14825 log.cc:1196] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/86397d343a29436eab33387e398fb951
I20260620 23:32:05.290268 14825 ts_tablet_manager.cc:1939] T 86397d343a29436eab33387e398fb951 P 6eee1b37e4de40bb9c43e2eb26762cb9: Deleting consensus metadata
I20260620 23:32:05.292621 14826 tablet_replica.cc:330] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: stopping tablet replica
I20260620 23:32:05.292557 14311 catalog_manager.cc:4841] TS 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): tablet 86397d343a29436eab33387e398fb951 (table test-table [id=6448a61bbbcc44bbbd4d5aa9b9e8a252]) successfully deleted
I20260620 23:32:05.293258 14826 raft_consensus.cc:2229] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:05.293836 14826 raft_consensus.cc:2258] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 20 23:32:05 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:32:05.295264 14824 ts_tablet_manager.cc:1918] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:05.295624 14824 log.cc:1196] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/86397d343a29436eab33387e398fb951
I20260620 23:32:05.296128 14826 ts_tablet_manager.cc:1905] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:05.296419 14824 ts_tablet_manager.cc:1939] T 86397d343a29436eab33387e398fb951 P d24a79d690ed4a25a24d9cef0e4edde6: Deleting consensus metadata
I20260620 23:32:05.299633 14311 catalog_manager.cc:4841] TS d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193:42903): tablet 86397d343a29436eab33387e398fb951 (table test-table [id=6448a61bbbcc44bbbd4d5aa9b9e8a252]) successfully deleted
I20260620 23:32:05.303730 14826 ts_tablet_manager.cc:1918] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:05.303967 14826 log.cc:1196] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/86397d343a29436eab33387e398fb951
I20260620 23:32:05.304385 14799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.267577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.0.96.193:42903)
0620 23:32:05.267815 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.267824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.280286 (+ 12462us) server_negotiation.cc:407] Connection header received
0620 23:32:05.280645 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.280654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.280763 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.280869 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:05.283458 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.284801 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.287035 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.287895 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.288013 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.296193 (+ 8180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:05.296210 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:05.296237 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:05.296306 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:05.300778 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:05.301695 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:05.301704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:05.301708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:05.301889 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:05.302417 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:05.302424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:05.302431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:05.302843 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:05.303199 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.303545 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.303821 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:32:05.304553 14826 ts_tablet_manager.cc:1939] T 86397d343a29436eab33387e398fb951 P e26660f20cae49019671ba389347835f: Deleting consensus metadata
I20260620 23:32:05.305255 14508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:05.306699 14308 catalog_manager.cc:4841] TS e26660f20cae49019671ba389347835f (127.0.96.194:34257): tablet 86397d343a29436eab33387e398fb951 (table test-table [id=6448a61bbbcc44bbbd4d5aa9b9e8a252]) successfully deleted
I20260620 23:32:05.409322 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14385
I20260620 23:32:05.434090 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14519
Jun 20 23:32:05 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:32:05.457391 14797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.435051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.0.96.194:34257)
0620 23:32:05.435273 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.435277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.435298 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:32:05.435500 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.435504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.435580 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.435686 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:05.437382 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.438172 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.439724 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.440214 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.440329 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.450295 (+ 9966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:05.450319 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:05.450341 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:05.450406 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:05.453852 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:05.454770 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:05.454780 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:05.454783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:05.455064 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:05.455480 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:05.455485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:05.455487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:05.456037 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:05.456364 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.456635 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.456864 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:32:05.458261 14642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:05.561513 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14519
I20260620 23:32:05.579813 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14653
Jun 20 23:32:05 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:32:05.597509 14796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:05.580736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.96.195:38007)
0620 23:32:05.580919 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:05.580923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:05.581102 (+ 179us) server_negotiation.cc:407] Connection header received
0620 23:32:05.581233 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:05.581237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:05.581314 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:05.581427 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:05.582809 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.583604 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.585087 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:05.585480 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:05.585597 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:05.590596 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:05.590616 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:05.590634 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:05.590676 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:05.594222 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:05.595269 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:05.595276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:05.595278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:05.595423 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:05.595933 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:05.595937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:05.595939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:05.596321 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:05.596567 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:05.596931 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 23:32:05.597241 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 23:32:05.598423 14776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:05.691162 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14653
I20260620 23:32:05.709219 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14292
I20260620 23:32:05.710698 14354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:05.825160 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14292
I20260620 23:32:05.849236 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35269
--webserver_interface=127.0.96.254
--webserver_port=35437
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:06.201941 14832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:06.202314 14832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:06.202457 14832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:06.210309 14832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:32:06.210453 14832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:06.210561 14832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:06.210644 14832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:32:06.210695 14832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:32:06.221486 14832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=35437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:06.222891 14832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:06.224261 14832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:06.251439 14838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:06.251519 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:06.251474 14839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:06.252720 14832 server_base.cc:1034] running on GCE node
I20260620 23:32:06.253824 14832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:06.255654 14832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:06.257052 14832 hybrid_clock.cc:648] HybridClock initialized: now 1781998326256943 us; error 58 us; skew 500 ppm
I20260620 23:32:06.257611 14832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:06 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:06.265117 14832 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:32:06.267365 14832 webserver.cc:460] Webserver started at http://127.0.96.254:35437/ using document root <none> and password file <none>
I20260620 23:32:06.268236 14832 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:06.268419 14832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:06.275796 14832 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20260620 23:32:06.279978 14848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:06.281970 14832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:32:06.282227 14832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "f7225466d04a4092afb14e25a3fbd8cb"
format_stamp: "Formatted at 2026-06-20 23:32:02 on dist-test-slave-v34r"
server_key: "4643cca18a9f09ff6e10311eca76469b"
server_key_iv: "399ce63d395b7dc7153719f844d1b82d"
server_key_version: "encryptionkey@0"
I20260620 23:32:06.283450 14832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:06.308666 14832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:06.313951 14832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:06.314335 14832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:06.338405 14832 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:35269
I20260620 23:32:06.338459 14899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:35269 every 8 connection(s)
I20260620 23:32:06.340143 14832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:32:06.340808 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14832
I20260620 23:32:06.342060 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:42903
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:35269
--webserver_port=39049
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:32:06.351694 14900 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:32:06.356576 14900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap starting.
I20260620 23:32:06.382965 14900 log.cc:824] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:06.401401 14900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:32:06.402163 14900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap complete.
I20260620 23:32:06.415071 14900 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:06.415999 14900 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7225466d04a4092afb14e25a3fbd8cb, State: Initialized, Role: FOLLOWER
I20260620 23:32:06.417274 14900 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:06.417656 14900 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:32:06.417829 14900 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:32:06.418087 14900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:32:06.423655 14900 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:06.424489 14900 leader_election.cc:302] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7225466d04a4092afb14e25a3fbd8cb; no voters:
I20260620 23:32:06.425434 14900 leader_election.cc:288] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:32:06.425601 14904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:32:06.427160 14904 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 LEADER]: Becoming Leader. State: Replica: f7225466d04a4092afb14e25a3fbd8cb, State: Running, Role: LEADER
I20260620 23:32:06.427948 14904 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:06.428555 14900 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:32:06.431692 14906 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7225466d04a4092afb14e25a3fbd8cb. Latest consensus state: current_term: 2 leader_uuid: "f7225466d04a4092afb14e25a3fbd8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } } }
I20260620 23:32:06.431784 14905 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f7225466d04a4092afb14e25a3fbd8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } } }
I20260620 23:32:06.432265 14906 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:06.432365 14905 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:06.434706 14911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:32:06.460228 14911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:32:06.462860 14911 catalog_manager.cc:1259] Loaded cluster ID: 42f06effb09a4b959a34f9378f3bec0a
I20260620 23:32:06.463057 14911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:32:06.468516 14911 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 23:32:06.503161 14911 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 23:32:06.530021 14911 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:32:06.532220 14911 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Loaded TSK: 0
I20260620 23:32:06.534121 14911 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:32:06.756595 14902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:06.757103 14902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:06.757288 14902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:06.766175 14902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:06.766338 14902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:06.766552 14902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:32:06.778386 14902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:42903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=39049
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:06.780270 14902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:06.782253 14902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:06.801422 14927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:06.801431 14928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:06.801714 14930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:06.803902 14902 server_base.cc:1034] running on GCE node
I20260620 23:32:06.804904 14902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:06.806370 14902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:06.807642 14902 hybrid_clock.cc:648] HybridClock initialized: now 1781998326807575 us; error 75 us; skew 500 ppm
I20260620 23:32:06.808087 14902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:06 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:06.812505 14902 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:32:06.814929 14902 webserver.cc:460] Webserver started at http://127.0.96.193:39049/ using document root <none> and password file <none>
I20260620 23:32:06.815966 14902 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:06.816174 14902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:06.823746 14902 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:32:06.827695 14937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:06.829872 14902 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:32:06.830133 14902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "d24a79d690ed4a25a24d9cef0e4edde6"
format_stamp: "Formatted at 2026-06-20 23:32:03 on dist-test-slave-v34r"
server_key: "6a64dd5abe9f0a3e1736f1f5908f846e"
server_key_iv: "a50f6e867b46f429858f12ad65e5a666"
server_key_version: "encryptionkey@0"
I20260620 23:32:06.831444 14902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:06.846742 14902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:06.852660 14902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:06.853112 14902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:06.854526 14902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:06.856361 14902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:06.856472 14902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:06.856665 14902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:06.856778 14902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:06.909206 14902 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:42903
I20260620 23:32:06.909363 15049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:42903 every 8 connection(s)
I20260620 23:32:06.911201 14902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:32:06.919217 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 14902
I20260620 23:32:06.920047 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:34257
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:35269
--webserver_port=35695
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:32:06 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:06.947031 15053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:06.915018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:42249)
0620 23:32:06.916102 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:06.916169 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:06.918338 (+ 2169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:06.920758 (+ 2420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:06.920769 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:06.921670 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:06.922398 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:06.922421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:06.925292 (+ 2871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:06.925302 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:06.927404 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:06.927423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:06.929481 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:06.929493 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:06.930052 (+ 559us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:06.931477 (+ 1425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:06.931552 (+ 75us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:06.934870 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:06.940611 (+ 5741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:06.940621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:06.940636 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:06.941366 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:06.942159 (+ 793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:06.942165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:06.942168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:06.942538 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:06.943916 (+ 1378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:06.943926 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:06.944496 (+ 570us) client_negotiation.cc:769] Sending connection context
0620 23:32:06.944845 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 23:32:06.945588 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":489,"threads_started":1}
I20260620 23:32:06.947583 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:06.914685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:42249 (local address 127.0.96.254:35269)
0620 23:32:06.915324 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:06.915337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:06.916359 (+ 1022us) server_negotiation.cc:407] Connection header received
0620 23:32:06.919169 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:06.919200 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:06.919829 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:06.920409 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:06.922712 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:06.924955 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:06.927613 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:06.928966 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:06.929240 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:06.935388 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:06.935403 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:06.935434 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:06.935508 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:06.940313 (+ 4805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:06.941594 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:06.941607 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:06.941618 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:06.941883 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:06.942728 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:06.942736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:06.942741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:06.943584 (+ 843us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:06.943817 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:06.945304 (+ 1487us) server_negotiation.cc:299] Negotiation successful
0620 23:32:06.945941 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":298,"threads_started":1}
I20260620 23:32:06.951579 15050 heartbeater.cc:347] Connected to a master server at 127.0.96.254:35269
I20260620 23:32:06.952044 15050 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:06.953126 15050 heartbeater.cc:511] Master 127.0.96.254:35269 requested a full tablet report, sending...
I20260620 23:32:06.956210 14865 ts_manager.cc:194] Registered new tserver with Master: d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193:42903)
I20260620 23:32:06.958169 14865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:42249
W20260620 23:32:07.271891 15054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:07.272266 15054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:07.272444 15054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:07.280049 15054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:07.280221 15054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:07.280418 15054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:32:07.291414 15054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:34257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=35695
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:07.293170 15054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:07.294580 15054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:07.309500 15059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:07.309836 15062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:07.309855 15060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:07.311184 15054 server_base.cc:1034] running on GCE node
I20260620 23:32:07.311865 15054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:07.313105 15054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:07.314380 15054 hybrid_clock.cc:648] HybridClock initialized: now 1781998327314328 us; error 52 us; skew 500 ppm
I20260620 23:32:07.314704 15054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:07 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:07.318781 15054 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:32:07.320672 15054 webserver.cc:460] Webserver started at http://127.0.96.194:35695/ using document root <none> and password file <none>
I20260620 23:32:07.321530 15054 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:07.321707 15054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:07.328476 15054 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:32:07.332109 15069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:07.333983 15054 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 23:32:07.334216 15054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "e26660f20cae49019671ba389347835f"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "ef6e8870c50384ffbd14218606ea70e7"
server_key_iv: "fd33b3854881e4e31196f3880b84417a"
server_key_version: "encryptionkey@0"
I20260620 23:32:07.335343 15054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:07.373317 15054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:07.379410 15054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:07.379966 15054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:07.381436 15054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:07.383347 15054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:07.383512 15054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:07.383764 15054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:07.383878 15054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:07.429426 15054 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34257
I20260620 23:32:07.429658 15181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34257 every 8 connection(s)
I20260620 23:32:07.431145 15054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:32:07.432932 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15054
I20260620 23:32:07.433754 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:38007
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:35269
--webserver_port=37987
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:32:07 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:07.470955 15052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:07.435140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:58929 (local address 127.0.96.254:35269)
0620 23:32:07.435449 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:07.435458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:07.442187 (+ 6729us) server_negotiation.cc:407] Connection header received
0620 23:32:07.445762 (+ 3575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:07.445769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:07.445934 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:07.446080 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:07.448777 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.449699 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.452893 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.453560 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.453751 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:07.461857 (+ 8106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:07.461866 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:07.461870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:07.462049 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:07.466106 (+ 4057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:07.467533 (+ 1427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:07.467540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:07.467543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:07.467779 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:07.468539 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:07.468544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:07.468547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:07.468967 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:07.469237 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:07.470321 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0620 23:32:07.470634 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260620 23:32:07.472208 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:07.435810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:58929)
0620 23:32:07.441866 (+ 6056us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:07.441929 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:07.445186 (+ 3257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:07.446469 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:07.446486 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:07.447450 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:07.448381 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:07.448413 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.449969 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.449978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:07.452505 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:07.452532 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.453939 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.453948 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:07.454345 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:07.455792 (+ 1447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:07.455843 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:07.461265 (+ 5422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:07.466406 (+ 5141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:07.466419 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:07.466442 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:07.467209 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:07.468011 (+ 802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:07.468017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:07.468022 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:07.468322 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:07.469249 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:07.469259 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:07.469781 (+ 522us) client_negotiation.cc:769] Sending connection context
0620 23:32:07.470131 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 23:32:07.470757 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5164,"thread_start_us":4819,"threads_started":1}
I20260620 23:32:07.474745 15182 heartbeater.cc:347] Connected to a master server at 127.0.96.254:35269
I20260620 23:32:07.475237 15182 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:07.476254 15182 heartbeater.cc:511] Master 127.0.96.254:35269 requested a full tablet report, sending...
I20260620 23:32:07.478583 14865 ts_manager.cc:194] Registered new tserver with Master: e26660f20cae49019671ba389347835f (127.0.96.194:34257)
I20260620 23:32:07.479629 14865 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:58929
W20260620 23:32:07.816483 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:07.816900 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:07.817095 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:07.825311 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:07.825480 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:07.825661 15185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:32:07.836274 15185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:38007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=37987
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:07.837925 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:07.839748 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:07.855415 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:07.855472 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:07.855913 15185 server_base.cc:1034] running on GCE node
W20260620 23:32:07.856958 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:07.857981 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:07.859369 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:07.860671 15185 hybrid_clock.cc:648] HybridClock initialized: now 1781998327860608 us; error 58 us; skew 500 ppm
I20260620 23:32:07.861146 15185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:07 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:07.865509 15185 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:32:07.867847 15185 webserver.cc:460] Webserver started at http://127.0.96.195:37987/ using document root <none> and password file <none>
I20260620 23:32:07.868666 15185 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:07.868855 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:07.875689 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 23:32:07.879554 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:07.881863 15185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:32:07.882184 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "965f5613608db0cab6e8b682e3d871c3"
server_key_iv: "12e3441a4664b41a2a0b77b8d0d47778"
server_key_version: "encryptionkey@0"
I20260620 23:32:07.883520 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:07.904701 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:07.910365 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:07.910738 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:07.913501 15185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:07.915249 15185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:07.915377 15185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:07.915570 15185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:07.915648 15185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:07.962806 15050 heartbeater.cc:503] Master 127.0.96.254:35269 was elected leader, sending a full tablet report...
I20260620 23:32:07.968245 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:38007
I20260620 23:32:07.968302 15312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:38007 every 8 connection(s)
I20260620 23:32:07.970372 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:32:07.977916 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15185
Jun 20 23:32:07 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:08.005800 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:07.979242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.96.254:35269)
0620 23:32:07.980031 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:07.980037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:07.981540 (+ 1503us) server_negotiation.cc:407] Connection header received
0620 23:32:07.982076 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:07.982081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:07.982190 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:07.982298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:07.985403 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.986909 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.989461 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.990105 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.990246 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:07.994238 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:07.994243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:07.994246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:07.994289 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:07.999771 (+ 5482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.001482 (+ 1711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.001490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.001495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.001887 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.002888 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.002895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.002899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.003631 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:08.003973 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.004832 (+ 859us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.005209 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":617,"spinlock_wait_cycles":21632,"thread_start_us":428,"threads_started":1}
I20260620 23:32:08.006124 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:07.974473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:38589 (local address 127.0.96.254:35269)
0620 23:32:07.975410 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:07.975415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:07.976899 (+ 1484us) server_negotiation.cc:407] Connection header received
0620 23:32:07.979373 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:07.979384 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:07.979581 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:07.979799 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:07.981995 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.983478 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.986737 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.987630 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.987851 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:07.994190 (+ 6339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:07.994199 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:07.994204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:07.994322 (+ 118us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:07.999803 (+ 5481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.001482 (+ 1679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.001490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.001495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.001887 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.002888 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.002895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.002899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.003631 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:08.003881 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.005307 (+ 1426us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.005524 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":785,"spinlock_wait_cycles":94848,"thread_start_us":628,"threads_started":1}
I20260620 23:32:08.007550 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:07.975537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:38589)
0620 23:32:07.976657 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:07.976720 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:07.978898 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:07.980220 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:07.980231 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:07.980871 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:07.981705 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:07.981728 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.983849 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.983855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:07.986401 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:07.986425 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:07.988068 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:07.988080 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:07.988460 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:07.989674 (+ 1214us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:07.989717 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:07.993659 (+ 3942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:08.000093 (+ 6434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:08.000106 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:08.000137 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:08.001236 (+ 1099us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:08.002143 (+ 907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:08.002150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:08.002155 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:08.002598 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:08.003863 (+ 1265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:08.003880 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:08.004709 (+ 829us) client_negotiation.cc:769] Sending connection context
0620 23:32:08.005188 (+ 479us) client_negotiation.cc:240] Negotiation successful
0620 23:32:08.005873 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":432,"threads_started":1}
I20260620 23:32:08.011013 15313 heartbeater.cc:347] Connected to a master server at 127.0.96.254:35269
I20260620 23:32:08.011451 15313 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:08.012912 15313 heartbeater.cc:511] Master 127.0.96.254:35269 requested a full tablet report, sending...
I20260620 23:32:08.015268 14864 ts_manager.cc:194] Registered new tserver with Master: 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
I20260620 23:32:08.016714 14864 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:38589
I20260620 23:32:08.026054 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:32:08.037771 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.027775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.0.96.254:35269)
0620 23:32:08.028366 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.028375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.028721 (+ 346us) server_negotiation.cc:407] Connection header received
0620 23:32:08.029625 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.029631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.029849 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.030013 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:32:08.030908 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.032108 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.034379 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.034919 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.035073 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.035930 (+ 857us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:32:08.036696 (+ 766us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:32:08.036872 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.037222 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.037317 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260620 23:32:08.038980 14864 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:32:08.048187 14864 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:32:08.084715 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.067232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34257 (local address 127.0.0.1:60168)
0620 23:32:08.068519 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.068602 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:08.069122 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:08.070261 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:08.070270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:08.070325 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:08.070893 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.070909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.074261 (+ 3352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.074272 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.078198 (+ 3926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.078244 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.081711 (+ 3467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.081724 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.082253 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.083307 (+ 1054us) client_negotiation.cc:769] Sending connection context
0620 23:32:08.083569 (+ 262us) client_negotiation.cc:240] Negotiation successful
0620 23:32:08.083984 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":934,"thread_start_us":1699,"threads_started":1}
I20260620 23:32:08.087318 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.067767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.0.96.194:34257)
0620 23:32:08.068445 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.068460 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.068817 (+ 357us) server_negotiation.cc:407] Connection header received
0620 23:32:08.069550 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.069560 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.069945 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.070232 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:08.071152 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.073950 (+ 2798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.078698 (+ 4748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.081234 (+ 2536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.082351 (+ 1117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.083097 (+ 746us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.086700 (+ 3603us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.086954 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":884,"threads_started":1}
I20260620 23:32:08.087886 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.068357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.0.96.193:42903)
0620 23:32:08.069174 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.069184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.070776 (+ 1592us) server_negotiation.cc:407] Connection header received
0620 23:32:08.070910 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.070919 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.072174 (+ 1255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.072456 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:08.073752 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.077207 (+ 3455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.082746 (+ 5539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.085180 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.085404 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.085675 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.087336 (+ 1661us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.087527 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":1140,"threads_started":1}
I20260620 23:32:08.088074 15325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.067542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59880)
0620 23:32:08.070413 (+ 2871us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.070440 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:08.070599 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:08.072837 (+ 2238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:08.072845 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:08.072910 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:08.073438 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.073454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.077525 (+ 4071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.077536 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.082345 (+ 4809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.082379 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.086856 (+ 4477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.086862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.087072 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.087139 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 23:32:08.087256 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:32:08.087419 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2708,"mutex_wait_us":99,"thread_start_us":2389,"threads_started":1}
I20260620 23:32:08.092523 14985 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8422a96e5e47968b849d93ba018e1d (DEFAULT_TABLE table=test-table [id=80ffe4cd54944ca494ced607cddbc485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:08.092839 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8422a96e5e47968b849d93ba018e1d (DEFAULT_TABLE table=test-table [id=80ffe4cd54944ca494ced607cddbc485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:08.094882 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8422a96e5e47968b849d93ba018e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:08.094916 14985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8422a96e5e47968b849d93ba018e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:08.097795 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.071008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39842 (local address 127.0.96.195:38007)
0620 23:32:08.086300 (+ 15292us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.086311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.086350 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:32:08.087054 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.087064 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.087430 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.087697 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:08.088459 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.090816 (+ 2357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.093916 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.096229 (+ 2313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.096462 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.096752 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.097101 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.097344 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15062,"thread_start_us":15697,"threads_started":1}
I20260620 23:32:08.097852 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.071293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:38007 (local address 127.0.0.1:39842)
0620 23:32:08.086242 (+ 14949us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.086265 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:08.086400 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:08.088027 (+ 1627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:08.088031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:08.088065 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:08.088268 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.088279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.091112 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.091120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.093314 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.093341 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.096522 (+ 3181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.096532 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.096761 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.096803 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 23:32:08.096961 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 23:32:08.097311 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14828,"thread_start_us":15256,"threads_started":1}
I20260620 23:32:08.104116 15248 tablet_service.cc:1467] Processing CreateTablet for tablet 2c8422a96e5e47968b849d93ba018e1d (DEFAULT_TABLE table=test-table [id=80ffe4cd54944ca494ced607cddbc485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:08.106580 15248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c8422a96e5e47968b849d93ba018e1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:08.114319 15330 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Bootstrap starting.
I20260620 23:32:08.118170 15330 tablet_bootstrap.cc:654] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:08.119313 15331 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Bootstrap starting.
I20260620 23:32:08.119691 15330 log.cc:824] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:08.124079 15331 tablet_bootstrap.cc:654] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:08.125314 15330 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: No bootstrap required, opened a new log
I20260620 23:32:08.125660 15330 ts_tablet_manager.cc:1397] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260620 23:32:08.125680 15331 log.cc:824] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:08.128791 15334 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Bootstrap starting.
I20260620 23:32:08.130330 15331 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: No bootstrap required, opened a new log
I20260620 23:32:08.130825 15331 ts_tablet_manager.cc:1397] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260620 23:32:08.133352 15334 tablet_bootstrap.cc:654] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:08.134822 15334 log.cc:824] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:08.137502 15330 raft_consensus.cc:348] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.138252 15330 raft_consensus.cc:374] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:08.138372 15330 raft_consensus.cc:729] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24a79d690ed4a25a24d9cef0e4edde6, State: Initialized, Role: FOLLOWER
I20260620 23:32:08.139425 15330 consensus_queue.cc:260] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.140208 15334 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: No bootstrap required, opened a new log
I20260620 23:32:08.140671 15334 ts_tablet_manager.cc:1397] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260620 23:32:08.142302 15330 ts_tablet_manager.cc:1428] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20260620 23:32:08.143337 15331 raft_consensus.cc:348] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.143926 15331 raft_consensus.cc:374] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:08.144052 15331 raft_consensus.cc:729] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26660f20cae49019671ba389347835f, State: Initialized, Role: FOLLOWER
I20260620 23:32:08.145491 15331 consensus_queue.cc:260] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.147346 15182 heartbeater.cc:503] Master 127.0.96.254:35269 was elected leader, sending a full tablet report...
I20260620 23:32:08.148200 15331 ts_tablet_manager.cc:1428] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
I20260620 23:32:08.155683 15334 raft_consensus.cc:348] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.156260 15334 raft_consensus.cc:374] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:08.156337 15334 raft_consensus.cc:729] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eee1b37e4de40bb9c43e2eb26762cb9, State: Initialized, Role: FOLLOWER
I20260620 23:32:08.157184 15334 consensus_queue.cc:260] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.159126 15313 heartbeater.cc:503] Master 127.0.96.254:35269 was elected leader, sending a full tablet report...
I20260620 23:32:08.160053 15334 ts_tablet_manager.cc:1428] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Time spent starting tablet: real 0.019s user 0.017s sys 0.002s
W20260620 23:32:08.164377 15051 tablet.cc:2369] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:32:08.183768 15183 tablet.cc:2369] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:32:08.222674 15314 tablet.cc:2369] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:08.260166 15337 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:08.260587 15337 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.262537 15337 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193:42903), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
I20260620 23:32:08.271595 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.263465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:43667 (local address 127.0.96.193:42903)
0620 23:32:08.263780 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.263790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.263951 (+ 161us) server_negotiation.cc:407] Connection header received
0620 23:32:08.264511 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.264518 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.264637 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.264714 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:08.265811 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.266756 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.269551 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.270490 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.270779 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.270869 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.271175 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.271317 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260620 23:32:08.271759 15339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.262941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.96.194:43667)
0620 23:32:08.263786 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.263811 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:08.264166 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:08.265024 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:08.265031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:08.265070 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:08.265595 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.265606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.266915 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.266924 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.269346 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.269362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.270839 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.270845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.271004 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.271030 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:32:08.271148 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:32:08.271291 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":33,"thread_start_us":496,"threads_started":1}
I20260620 23:32:08.272462 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.264438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:48549 (local address 127.0.96.195:38007)
0620 23:32:08.264719 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.264725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.264748 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:32:08.264835 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.264840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.264925 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.265082 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:08.266010 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.267174 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.269850 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.271076 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.271384 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.271515 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.271978 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.272150 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 23:32:08.273370 15340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.263956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:38007 (local address 127.0.96.194:48549)
0620 23:32:08.264608 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.264624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:08.264732 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:08.265191 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:08.265200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:08.265237 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:08.265803 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.265819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.267351 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.267359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.269641 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:08.269654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.271462 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.271471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:08.271714 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.271758 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 23:32:08.272734 (+ 976us) client_negotiation.cc:240] Negotiation successful
0620 23:32:08.272952 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":347,"threads_started":1}
I20260620 23:32:08.273144 15005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8422a96e5e47968b849d93ba018e1d" candidate_uuid: "e26660f20cae49019671ba389347835f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" is_pre_election: true
I20260620 23:32:08.273931 15005 raft_consensus.cc:2454] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26660f20cae49019671ba389347835f in term 0.
I20260620 23:32:08.274827 15073 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6, e26660f20cae49019671ba389347835f; no voters:
I20260620 23:32:08.274750 15268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8422a96e5e47968b849d93ba018e1d" candidate_uuid: "e26660f20cae49019671ba389347835f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" is_pre_election: true
I20260620 23:32:08.275352 15337 raft_consensus.cc:2781] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:32:08.275399 15268 raft_consensus.cc:2454] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26660f20cae49019671ba389347835f in term 0.
I20260620 23:32:08.275542 15337 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:32:08.275688 15337 raft_consensus.cc:3037] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:08.278250 15337 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.278980 15337 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [CANDIDATE]: Term 1 election: Requested vote from peers d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193:42903), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
I20260620 23:32:08.279742 15268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8422a96e5e47968b849d93ba018e1d" candidate_uuid: "e26660f20cae49019671ba389347835f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
I20260620 23:32:08.279881 15005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c8422a96e5e47968b849d93ba018e1d" candidate_uuid: "e26660f20cae49019671ba389347835f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d24a79d690ed4a25a24d9cef0e4edde6"
I20260620 23:32:08.280095 15268 raft_consensus.cc:3037] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:08.280185 15005 raft_consensus.cc:3037] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:08.282694 15005 raft_consensus.cc:2454] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26660f20cae49019671ba389347835f in term 1.
I20260620 23:32:08.283092 15268 raft_consensus.cc:2454] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26660f20cae49019671ba389347835f in term 1.
I20260620 23:32:08.283363 15073 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6, e26660f20cae49019671ba389347835f; no voters:
I20260620 23:32:08.283769 15337 raft_consensus.cc:2781] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:08.284365 15337 raft_consensus.cc:686] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 1 LEADER]: Becoming Leader. State: Replica: e26660f20cae49019671ba389347835f, State: Running, Role: LEADER
I20260620 23:32:08.285102 15337 consensus_queue.cc:237] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:08.291977 14865 catalog_manager.cc:5495] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f reported cstate change: term changed from 0 to 1, leader changed from <none> to e26660f20cae49019671ba389347835f (127.0.96.194). New cstate: current_term: 1 leader_uuid: "e26660f20cae49019671ba389347835f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } health_report { overall_health: UNKNOWN } } }
I20260620 23:32:08.330603 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.324832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.0.96.194:34257)
0620 23:32:08.325146 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.325153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.325176 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:32:08.325435 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.325440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.325522 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.325745 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:32:08.326124 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.326831 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.328291 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.328704 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.328839 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.329453 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:32:08.329979 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:32:08.330169 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.330221 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.330304 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:32:08.341866 15005 raft_consensus.cc:1261] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Refusing update from remote peer e26660f20cae49019671ba389347835f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:08.341907 15268 raft_consensus.cc:1261] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Refusing update from remote peer e26660f20cae49019671ba389347835f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:08.343261 15341 consensus_queue.cc:1035] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:08.343861 15337 consensus_queue.cc:1035] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:08.362597 15344 mvcc.cc:205] Tried to move back new op lower bound from 7299065152870039552 to 7299065152662630400. Current Snapshot: MvccSnapshot[applied={T|T < 7299065152870039552}]
I20260620 23:32:08.386066 14864 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:08.386345 14864 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:08.391088 14864 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Sending DeleteTablet for 3 replicas of tablet 2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:08.392807 15248 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8422a96e5e47968b849d93ba018e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:08 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:39842
I20260620 23:32:08.392807 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8422a96e5e47968b849d93ba018e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:08 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:60168
I20260620 23:32:08.393042 14985 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c8422a96e5e47968b849d93ba018e1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:08 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:59880
I20260620 23:32:08.394448 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14902
I20260620 23:32:08.408708 15354 tablet_replica.cc:330] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: stopping tablet replica
I20260620 23:32:08.409476 15354 raft_consensus.cc:2229] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:32:08.410686 15356 tablet_replica.cc:330] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: stopping tablet replica
I20260620 23:32:08.411649 15356 raft_consensus.cc:2229] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:08.412370 15356 raft_consensus.cc:2258] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:08.415370 15354 raft_consensus.cc:2258] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:08.415863 15355 tablet_replica.cc:330] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: stopping tablet replica
I20260620 23:32:08.416414 15355 raft_consensus.cc:2229] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:08.417034 15355 raft_consensus.cc:2258] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:08.417411 15356 ts_tablet_manager.cc:1905] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:08.419251 15354 ts_tablet_manager.cc:1905] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:08.419744 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.395786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.0.96.193:42903)
0620 23:32:08.396000 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.396008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.396031 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:32:08.396230 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.396235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.396385 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.396539 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:08.399152 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.400888 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.403715 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.404753 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.404915 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.410181 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:08.410213 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:08.410240 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:08.410315 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:08.414889 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.416240 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.416252 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.416255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.416489 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.417261 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.417267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.417270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.417818 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:08.418191 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.418960 (+ 769us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.419245 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 23:32:08.420737 15045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:08.422359 15355 ts_tablet_manager.cc:1905] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:08.427567 15354 ts_tablet_manager.cc:1918] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:08.427785 15354 log.cc:1196] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:08.428417 15354 ts_tablet_manager.cc:1939] T 2c8422a96e5e47968b849d93ba018e1d P e26660f20cae49019671ba389347835f: Deleting consensus metadata
I20260620 23:32:08.430760 14849 catalog_manager.cc:4841] TS e26660f20cae49019671ba389347835f (127.0.96.194:34257): tablet 2c8422a96e5e47968b849d93ba018e1d (table test-table [id=80ffe4cd54944ca494ced607cddbc485]) successfully deleted
I20260620 23:32:08.435025 15355 ts_tablet_manager.cc:1918] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:08.435233 15355 log.cc:1196] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:08.435770 15355 ts_tablet_manager.cc:1939] T 2c8422a96e5e47968b849d93ba018e1d P 6eee1b37e4de40bb9c43e2eb26762cb9: Deleting consensus metadata
I20260620 23:32:08.437779 14852 catalog_manager.cc:4841] TS 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): tablet 2c8422a96e5e47968b849d93ba018e1d (table test-table [id=80ffe4cd54944ca494ced607cddbc485]) successfully deleted
I20260620 23:32:08.546602 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14902
I20260620 23:32:08.566954 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 15054
W20260620 23:32:08.566911 14852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:42903 (error 108)
I20260620 23:32:08.569777 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.568917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59892)
0620 23:32:08.569307 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.569499 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 23:32:08.570199 14852 catalog_manager.cc:4584] TS d24a79d690ed4a25a24d9cef0e4edde6 (127.0.96.193:42903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c8422a96e5e47968b849d93ba018e1d: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
I20260620 23:32:08.588181 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.568598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.0.96.194:34257)
0620 23:32:08.568832 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.568840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.568867 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:32:08.568949 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.568954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.569304 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.569530 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:08.571670 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.572812 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.575080 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.575673 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.575852 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.579616 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:08.579639 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:08.579662 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:08.579721 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:08.583792 (+ 4071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.584858 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.584869 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.584872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.585188 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.585709 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.585713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.585716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.586222 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:08.586559 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.587191 (+ 632us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.587608 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:32:08.589234 15177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:08.624552 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.623788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59904)
0620 23:32:08.624140 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.624323 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 23:32:08.626066 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.625532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59908)
0620 23:32:08.625730 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.625841 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:32:08.687107 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.686379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59912)
0620 23:32:08.686716 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.686869 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 23:32:08.688769 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.688102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59928)
0620 23:32:08.688328 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.688535 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:32:08.723471 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 15054
I20260620 23:32:08.745780 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 15185
I20260620 23:32:08.762208 15328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.746703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.0.96.195:38007)
0620 23:32:08.746962 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:08.746970 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:08.747111 (+ 141us) server_negotiation.cc:407] Connection header received
0620 23:32:08.747369 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:08.747375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:08.747465 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:08.747585 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:08.749210 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.750032 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.751536 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:08.751978 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:08.752131 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:08.755089 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:08.755108 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:08.755126 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:08.755183 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:08.758709 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.759906 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.759914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.759917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.760093 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:08.760604 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:08.760607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:08.760609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:08.761034 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:08.761375 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:08.761645 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:32:08.761849 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 23:32:08.763145 15308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:08.770282 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.769531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59932)
0620 23:32:08.769898 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.770054 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 23:32:08.771874 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:08.771239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59942)
0620 23:32:08.771541 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:08.771654 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 23:32:08.867343 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 15185
I20260620 23:32:08.886654 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 14832
I20260620 23:32:08.888485 14895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:09.016455 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:09.015387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:42903 (local address 127.0.0.1:59958)
0620 23:32:09.015943 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:09.016082 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:42903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:32:09.016867 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 14832
I20260620 23:32:09.043249 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35269
--webserver_interface=127.0.96.254
--webserver_port=35437
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:35269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:09.420591 15362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:09.420914 15362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:09.421084 15362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:09.428783 15362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:32:09.428915 15362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:09.428963 15362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:09.429090 15362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:32:09.429167 15362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:32:09.439625 15362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:35269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=35437
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:09.441139 15362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:09.442479 15362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:09.455199 15367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:09.455358 15370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:09.455575 15368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:09.456583 15362 server_base.cc:1034] running on GCE node
I20260620 23:32:09.457654 15362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:09.459344 15362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:09.460681 15362 hybrid_clock.cc:648] HybridClock initialized: now 1781998329460585 us; error 85 us; skew 500 ppm
I20260620 23:32:09.461218 15362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:09 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:09.465298 15362 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:32:09.467293 15362 webserver.cc:460] Webserver started at http://127.0.96.254:35437/ using document root <none> and password file <none>
I20260620 23:32:09.468137 15362 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:09.468294 15362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:09.474928 15362 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:32:09.480494 15377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:09.483001 15362 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:32:09.483295 15362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "f7225466d04a4092afb14e25a3fbd8cb"
format_stamp: "Formatted at 2026-06-20 23:32:02 on dist-test-slave-v34r"
server_key: "4643cca18a9f09ff6e10311eca76469b"
server_key_iv: "399ce63d395b7dc7153719f844d1b82d"
server_key_version: "encryptionkey@0"
I20260620 23:32:09.484668 15362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:09.512506 15362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:09.518821 15362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:09.519222 15362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:09.543810 15362 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:35269
I20260620 23:32:09.543836 15428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:35269 every 8 connection(s)
I20260620 23:32:09.545852 15362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:32:09.548288 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15362
I20260620 23:32:09.549288 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:42903
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:35269
--webserver_port=39049
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:32:09.554641 15429 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:32:09.559372 15429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap starting.
I20260620 23:32:09.593343 15429 log.cc:824] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:09.615069 15429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:32:09.615825 15429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb: Bootstrap complete.
I20260620 23:32:09.627081 15429 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:09.628460 15429 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7225466d04a4092afb14e25a3fbd8cb, State: Initialized, Role: FOLLOWER
I20260620 23:32:09.629362 15429 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:09.629663 15429 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:32:09.629814 15429 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:32:09.630025 15429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:32:09.635622 15429 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:09.636390 15429 leader_election.cc:302] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7225466d04a4092afb14e25a3fbd8cb; no voters:
I20260620 23:32:09.637219 15429 leader_election.cc:288] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:32:09.637490 15433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:32:09.638923 15433 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [term 3 LEADER]: Becoming Leader. State: Replica: f7225466d04a4092afb14e25a3fbd8cb, State: Running, Role: LEADER
I20260620 23:32:09.639739 15433 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } }
I20260620 23:32:09.640012 15429 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:32:09.642951 15435 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7225466d04a4092afb14e25a3fbd8cb. Latest consensus state: current_term: 3 leader_uuid: "f7225466d04a4092afb14e25a3fbd8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } } }
I20260620 23:32:09.643530 15435 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:09.644527 15434 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f7225466d04a4092afb14e25a3fbd8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7225466d04a4092afb14e25a3fbd8cb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35269 } } }
I20260620 23:32:09.645099 15434 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7225466d04a4092afb14e25a3fbd8cb [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:09.655479 15439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:32:09.669351 15439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:32:09.671986 15439 catalog_manager.cc:1259] Loaded cluster ID: 42f06effb09a4b959a34f9378f3bec0a
I20260620 23:32:09.672278 15439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 23:32:09.676047 15439 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f30c4f019ad google::LogMessage::Fail() at ??:0
@ 0x7f30c4f0488a google::LogMessage::SendToLog() at ??:0
@ 0x7f30c4f014de google::LogMessage::Flush() at ??:0
@ 0x7f30c4f02a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f30da934512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f30da931976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f30da9b9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f30da9b9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f30c7b4d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f30c7b5091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f30c7b50799 std::_Function_handler<>::_M_invoke() at ??:0
I20260620 23:32:09.748903 15458 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f30c7b2beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f30cae056db start_thread at ??:0
@ 0x7f30c3d9b61f clone at ??:0
W20260620 23:32:09.996510 15431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:09.996835 15431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:09.996940 15431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:10.005633 15431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:10.005792 15431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:10.005999 15431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:32:10.017503 15431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:42903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=39049
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:10.019299 15431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:10.021749 15431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:10.044381 15473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:10.044368 15470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:10.044405 15471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:10.044930 15431 server_base.cc:1034] running on GCE node
I20260620 23:32:10.046375 15431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:10.047683 15431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:10.049060 15431 hybrid_clock.cc:648] HybridClock initialized: now 1781998330048926 us; error 80 us; skew 500 ppm
I20260620 23:32:10.049516 15431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:10 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:10.054257 15431 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:32:10.056308 15431 webserver.cc:460] Webserver started at http://127.0.96.193:39049/ using document root <none> and password file <none>
I20260620 23:32:10.057190 15431 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:10.057354 15431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:10.064050 15431 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:32:10.067879 15480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:10.069926 15431 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:32:10.070152 15431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "d24a79d690ed4a25a24d9cef0e4edde6"
format_stamp: "Formatted at 2026-06-20 23:32:03 on dist-test-slave-v34r"
server_key: "6a64dd5abe9f0a3e1736f1f5908f846e"
server_key_iv: "a50f6e867b46f429858f12ad65e5a666"
server_key_version: "encryptionkey@0"
I20260620 23:32:10.071556 15431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:10.090453 15431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:10.095083 15431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:10.095394 15431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:10.096554 15431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:10.099021 15487 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260620 23:32:10.107038 15431 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 23:32:10.107286 15431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.001s
I20260620 23:32:10.107466 15431 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 23:32:10.111212 15431 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 23:32:10.111352 15431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260620 23:32:10.112061 15487 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Bootstrap starting.
I20260620 23:32:10.125941 15487 log.cc:824] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:10.150283 15487 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:32:10.151151 15487 tablet_bootstrap.cc:492] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Bootstrap complete.
I20260620 23:32:10.151841 15487 ts_tablet_manager.cc:1397] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Time spent bootstrapping tablet: real 0.040s user 0.032s sys 0.004s
I20260620 23:32:10.162395 15431 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:42903
I20260620 23:32:10.162523 15594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:42903 every 8 connection(s)
I20260620 23:32:10.164319 15431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:32:10.166749 15487 raft_consensus.cc:348] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:10.167797 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15431
I20260620 23:32:10.167832 15487 raft_consensus.cc:729] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24a79d690ed4a25a24d9cef0e4edde6, State: Initialized, Role: FOLLOWER
I20260620 23:32:10.168635 15487 consensus_queue.cc:260] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:10.168706 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:34257
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:35269
--webserver_port=35695
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:32:10.173038 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:10.169375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:41131)
0620 23:32:10.171028 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:10.171461 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":703,"threads_started":1}
I20260620 23:32:10.173055 15487 ts_tablet_manager.cc:1428] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
W20260620 23:32:10.174336 15595 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:35269 (0 consecutive failures): Network error: Failed to ping master at 127.0.96.254:35269: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
I20260620 23:32:10.175930 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:10.175294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:38575)
0620 23:32:10.175499 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:10.175794 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:32:10.177579 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:10.176899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:57257)
0620 23:32:10.177167 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:10.177327 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 23:32:10.178011 15595 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:32:10.547776 15599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:10.548146 15599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:10.548314 15599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:10.555968 15599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:10.556154 15599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:10.556344 15599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:32:10.567401 15599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:34257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=35695
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:10.568858 15599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:10.570281 15599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:10.589558 15604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:10.589679 15607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:10.589645 15605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:10.591796 15599 server_base.cc:1034] running on GCE node
I20260620 23:32:10.592557 15599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:10.594027 15599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:10.595309 15599 hybrid_clock.cc:648] HybridClock initialized: now 1781998330595204 us; error 104 us; skew 500 ppm
I20260620 23:32:10.595682 15599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:10 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:10.599416 15599 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:32:10.601337 15599 webserver.cc:460] Webserver started at http://127.0.96.194:35695/ using document root <none> and password file <none>
I20260620 23:32:10.602123 15599 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:10.602262 15599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:10.608461 15599 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 23:32:10.612128 15614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:10.613915 15599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 23:32:10.614146 15599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "e26660f20cae49019671ba389347835f"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "ef6e8870c50384ffbd14218606ea70e7"
server_key_iv: "fd33b3854881e4e31196f3880b84417a"
server_key_version: "encryptionkey@0"
I20260620 23:32:10.615268 15599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:10.645998 15599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:10.651625 15599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:10.652002 15599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:10.653378 15599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:10.655032 15599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:10.655174 15599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:10.655325 15599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:10.655432 15599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:10.698701 15599 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34257
I20260620 23:32:10.698719 15726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34257 every 8 connection(s)
I20260620 23:32:10.700578 15599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:32:10.707423 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15599
I20260620 23:32:10.707558 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:10.704582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:49879)
0620 23:32:10.705706 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:10.706017 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":471,"threads_started":1}
I20260620 23:32:10.708293 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:38007
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:35269
--webserver_port=37987
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:10.708565 15727 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:35269 (0 consecutive failures): Network error: Failed to ping master at 127.0.96.254:35269: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
I20260620 23:32:10.709939 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:10.709313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:51845)
0620 23:32:10.709575 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:10.709713 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:32:10.711498 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:10.710993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:40635)
0620 23:32:10.711218 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:10.711326 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 23:32:10.711968 15727 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:32:11.095853 15730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:11.096231 15730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:11.096374 15730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:11.105119 15730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:11.105343 15730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:11.105562 15730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:32:11.116516 15730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:38007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=37987
--tserver_master_addrs=127.0.96.254:35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:11.118227 15730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:11.120043 15730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:11.135452 15735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:11.135730 15738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:11.135821 15736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:11.136889 15730 server_base.cc:1034] running on GCE node
I20260620 23:32:11.137681 15730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:11.138805 15730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:11.140134 15730 hybrid_clock.cc:648] HybridClock initialized: now 1781998331140065 us; error 66 us; skew 500 ppm
I20260620 23:32:11.140643 15730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:11 dist-test-slave-v34r krb5kdc[14279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:11.144898 15730 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:32:11.147146 15730 webserver.cc:460] Webserver started at http://127.0.96.195:37987/ using document root <none> and password file <none>
I20260620 23:32:11.147964 15730 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:11.148159 15730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:11.155790 15730 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:32:11.159875 15745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:11.162075 15730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:32:11.162346 15730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "6eee1b37e4de40bb9c43e2eb26762cb9"
format_stamp: "Formatted at 2026-06-20 23:32:04 on dist-test-slave-v34r"
server_key: "965f5613608db0cab6e8b682e3d871c3"
server_key_iv: "12e3441a4664b41a2a0b77b8d0d47778"
server_key_version: "encryptionkey@0"
I20260620 23:32:11.163496 15730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:11.179864 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.178808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:39517)
0620 23:32:11.179480 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.179612 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":353,"threads_started":1}
I20260620 23:32:11.184785 15730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:11.190135 15730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:11.190474 15730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:11.191720 15730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:11.193444 15730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:11.193550 15730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:11.193886 15730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:11.193998 15730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:11.237666 15730 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:38007
I20260620 23:32:11.237848 15858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:38007 every 8 connection(s)
I20260620 23:32:11.239306 15730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:32:11.240226 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15730
I20260620 23:32:11.246402 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.243666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:59831)
0620 23:32:11.244718 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.245103 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":401,"threads_started":1}
W20260620 23:32:11.247462 15859 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:35269 (0 consecutive failures): Network error: Failed to ping master at 127.0.96.254:35269: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
I20260620 23:32:11.248647 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.248213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:34605)
0620 23:32:11.248416 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.248513 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 23:32:11.250016 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.249598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:33111)
0620 23:32:11.249767 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.249862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:32:11.250422 15859 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:32:11.714233 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.712900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:59271)
0620 23:32:11.713720 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.713841 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":397,"threads_started":1}
I20260620 23:32:11.879614 15864 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:11.880026 15864 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:11.882047 15864 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26660f20cae49019671ba389347835f (127.0.96.194:34257), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
Jun 20 23:32:11 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
Jun 20 23:32:11 dist-test-slave-v34r krb5kdc[14279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:32:11.934257 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.882696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34257 (local address 127.0.96.193:52051)
0620 23:32:11.898454 (+ 15758us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.898476 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:11.898668 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:11.902803 (+ 4135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:11.902811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:11.903691 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:11.905059 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:11.905089 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.908434 (+ 3345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.908440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:11.909915 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:11.909934 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.912240 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.912252 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:11.912446 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:11.918655 (+ 6209us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:11.918681 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:11.923007 (+ 4326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:11.929034 (+ 6027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:11.929047 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:11.929072 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:11.929690 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:11.930712 (+ 1022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:11.930721 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:11.930725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:11.931025 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:11.932508 (+ 1483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:11.932520 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:11.933081 (+ 561us) client_negotiation.cc:769] Sending connection context
0620 23:32:11.933446 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 23:32:11.933923 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15628,"mutex_wait_us":129,"spinlock_wait_cycles":16768,"thread_start_us":14802,"threads_started":1}
I20260620 23:32:11.935108 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.882918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52051 (local address 127.0.96.194:34257)
0620 23:32:11.897464 (+ 14546us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:11.897491 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:11.898644 (+ 1153us) server_negotiation.cc:407] Connection header received
0620 23:32:11.900900 (+ 2256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:11.900912 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:11.901543 (+ 631us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:11.902009 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:11.906292 (+ 4283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.908147 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.910185 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.911564 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.912153 (+ 589us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:11.923680 (+ 11527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:11.923699 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:11.923738 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:11.923803 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:11.928667 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:11.930058 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:11.930074 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:11.930082 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:11.930428 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:11.931230 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:11.931238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:11.931243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:11.932324 (+ 1081us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:11.932534 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:11.933856 (+ 1322us) server_negotiation.cc:299] Negotiation successful
0620 23:32:11.934490 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14271,"thread_start_us":14070,"threads_started":1}
I20260620 23:32:11.935820 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.882646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:38007 (local address 127.0.96.193:59089)
0620 23:32:11.897372 (+ 14726us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:11.897478 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:11.898678 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:11.902206 (+ 3528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:11.902221 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:11.903696 (+ 1475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:11.905043 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:11.905090 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.907246 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.907251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:11.908766 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:11.908788 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.911049 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.911056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:11.911397 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:11.918626 (+ 7229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:11.918684 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:11.923453 (+ 4769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:11.932095 (+ 8642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:11.932099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:11.932102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:11.932648 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:11.933347 (+ 699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:11.933353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:11.933358 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:11.933540 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:11.934893 (+ 1353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:11.934900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:11.935055 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 23:32:11.935171 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 23:32:11.935519 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14500,"mutex_wait_us":69,"spinlock_wait_cycles":11776,"thread_start_us":15325,"threads_started":1}
I20260620 23:32:11.936715 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:11.883077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:59089 (local address 127.0.96.195:38007)
0620 23:32:11.898245 (+ 15168us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:11.898263 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:11.898331 (+ 68us) server_negotiation.cc:407] Connection header received
0620 23:32:11.900814 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:11.900825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:11.901400 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:11.902009 (+ 609us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:11.905405 (+ 3396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.906956 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.909036 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:11.910738 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:11.911028 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:11.923968 (+ 12940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:11.923978 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:11.923995 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:11.924050 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:11.931829 (+ 7779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:11.932850 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:11.932859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:11.932868 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:11.933194 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:11.933682 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:11.933687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:11.933690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:11.934479 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:11.934887 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:11.935533 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 23:32:11.936117 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14792,"thread_start_us":14620,"threads_started":1}
W20260620 23:32:11.940662 15484 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
W20260620 23:32:11.941342 15481 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e26660f20cae49019671ba389347835f (127.0.96.194:34257): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:11.941959 15481 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6; no voters: 6eee1b37e4de40bb9c43e2eb26762cb9, e26660f20cae49019671ba389347835f
I20260620 23:32:11.942929 15864 raft_consensus.cc:2726] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:32:12.181493 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:12.180962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:53069)
0620 23:32:12.181218 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:12.181343 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 23:32:12.252475 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:12.251205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:59893)
0620 23:32:12.251944 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:12.252145 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":299,"threads_started":1}
I20260620 23:32:12.716626 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:12.715530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:43987)
0620 23:32:12.716254 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:12.716397 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":375,"threads_started":1}
I20260620 23:32:13.183497 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:13.182645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:42823)
0620 23:32:13.183182 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:13.183306 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":281,"threads_started":1}
I20260620 23:32:13.254833 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:13.253914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:45559)
0620 23:32:13.254501 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:13.254615 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":287,"threads_started":1}
I20260620 23:32:13.719045 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:13.718021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:52329)
0620 23:32:13.718687 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:13.718815 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":395,"threads_started":1}
I20260620 23:32:13.769348 15874 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:13.769673 15874 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:13.770418 15874 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26660f20cae49019671ba389347835f (127.0.96.194:34257), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
W20260620 23:32:13.772027 15481 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e26660f20cae49019671ba389347835f (127.0.96.194:34257): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
W20260620 23:32:13.772320 15484 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:13.772455 15484 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6; no voters: 6eee1b37e4de40bb9c43e2eb26762cb9, e26660f20cae49019671ba389347835f
I20260620 23:32:13.772823 15874 raft_consensus.cc:2726] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:32:14.186071 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:14.184904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:49451)
0620 23:32:14.185671 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:14.185830 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":415,"threads_started":1}
I20260620 23:32:14.257673 15876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:14.256352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:41055)
0620 23:32:14.257085 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:14.257211 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":383,"threads_started":1}
I20260620 23:32:14.721494 15877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:14.720237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:58945)
0620 23:32:14.720951 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:14.721183 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":355,"threads_started":1}
I20260620 23:32:15.188760 15878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:15.187691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:45517)
0620 23:32:15.188421 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:15.188552 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":381,"threads_started":1}
I20260620 23:32:15.259961 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:15.259018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:37803)
0620 23:32:15.259647 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:15.259757 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":347,"threads_started":1}
I20260620 23:32:15.723802 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:15.722801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:60777)
0620 23:32:15.723435 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:15.723578 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":325,"threads_started":1}
I20260620 23:32:16.191124 15881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:16.190132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:44681)
0620 23:32:16.190796 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:16.190907 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":367,"threads_started":1}
I20260620 23:32:16.262202 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:16.261412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:44777)
0620 23:32:16.261947 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:16.262035 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":306,"threads_started":1}
I20260620 23:32:16.685189 15883 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:16.685423 15883 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:16.686241 15883 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26660f20cae49019671ba389347835f (127.0.96.194:34257), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
W20260620 23:32:16.687680 15481 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e26660f20cae49019671ba389347835f (127.0.96.194:34257): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
W20260620 23:32:16.687938 15484 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:16.688081 15484 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6; no voters: 6eee1b37e4de40bb9c43e2eb26762cb9, e26660f20cae49019671ba389347835f
I20260620 23:32:16.688382 15883 raft_consensus.cc:2726] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:32:16.726040 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:16.725105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:53619)
0620 23:32:16.725720 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:16.725829 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":325,"threads_started":1}
I20260620 23:32:17.193760 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:17.192626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:43841)
0620 23:32:17.193315 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:17.193440 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":386,"threads_started":1}
I20260620 23:32:17.264598 15886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:17.263671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:33459)
0620 23:32:17.264290 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:17.264394 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":345,"threads_started":1}
I20260620 23:32:17.728689 15887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:17.727232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:50545)
0620 23:32:17.728211 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:17.728354 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":494,"threads_started":1}
I20260620 23:32:18.196204 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:18.195158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:41123)
0620 23:32:18.195827 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:18.195953 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":356,"threads_started":1}
I20260620 23:32:18.267007 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:18.266121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:39115)
0620 23:32:18.266699 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:18.266793 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":278,"threads_started":1}
I20260620 23:32:18.459167 15890 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:18.459384 15890 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:18.460175 15890 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26660f20cae49019671ba389347835f (127.0.96.194:34257), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
W20260620 23:32:18.461699 15481 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e26660f20cae49019671ba389347835f (127.0.96.194:34257): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
W20260620 23:32:18.462026 15484 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:18.462250 15484 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6; no voters: 6eee1b37e4de40bb9c43e2eb26762cb9, e26660f20cae49019671ba389347835f
I20260620 23:32:18.462589 15890 raft_consensus.cc:2726] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:32:18.730952 15891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:18.730023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:40483)
0620 23:32:18.730632 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:18.730749 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":326,"threads_started":1}
I20260620 23:32:19.198822 15892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:19.197619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:50753)
0620 23:32:19.198457 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:19.198587 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":492,"threads_started":1}
I20260620 23:32:19.269701 15893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:19.268448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:36853)
0620 23:32:19.269265 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:19.269445 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":410,"threads_started":1}
I20260620 23:32:19.733525 15894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:19.732403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:49239)
0620 23:32:19.733165 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:19.733346 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":321,"threads_started":1}
I20260620 23:32:20.201632 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:20.200353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:41691)
0620 23:32:20.201283 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:20.201411 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":364,"threads_started":1}
I20260620 23:32:20.272118 15896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:20.271018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:42755)
0620 23:32:20.271697 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:20.271841 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":292,"threads_started":1}
I20260620 23:32:20.334048 15897 raft_consensus.cc:482] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:20.334295 15897 raft_consensus.cc:504] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26660f20cae49019671ba389347835f" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34257 } } peers { permanent_uuid: "d24a79d690ed4a25a24d9cef0e4edde6" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 42903 } } peers { permanent_uuid: "6eee1b37e4de40bb9c43e2eb26762cb9" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38007 } }
I20260620 23:32:20.335069 15897 leader_election.cc:288] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e26660f20cae49019671ba389347835f (127.0.96.194:34257), 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007)
W20260620 23:32:20.336601 15484 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6eee1b37e4de40bb9c43e2eb26762cb9 (127.0.96.195:38007): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
W20260620 23:32:20.336849 15481 leader_election.cc:341] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e26660f20cae49019671ba389347835f (127.0.96.194:34257): Not found: Tablet not found: 2c8422a96e5e47968b849d93ba018e1d
I20260620 23:32:20.337212 15481 leader_election.cc:302] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d24a79d690ed4a25a24d9cef0e4edde6; no voters: 6eee1b37e4de40bb9c43e2eb26762cb9, e26660f20cae49019671ba389347835f
I20260620 23:32:20.337590 15897 raft_consensus.cc:2726] T 2c8422a96e5e47968b849d93ba018e1d P d24a79d690ed4a25a24d9cef0e4edde6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 23:32:20.735838 15898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:20.734947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:44985)
0620 23:32:20.735545 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:20.735645 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":331,"threads_started":1}
I20260620 23:32:21.203831 15899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:21.202918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:51391)
0620 23:32:21.203506 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:21.203605 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":287,"threads_started":1}
I20260620 23:32:21.274890 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:21.273884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:34393)
0620 23:32:21.274561 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:21.274666 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":346,"threads_started":1}
I20260620 23:32:21.738337 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:21.737329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:46661)
0620 23:32:21.737981 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:21.738081 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":326,"threads_started":1}
I20260620 23:32:22.206545 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:22.205506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:50301)
0620 23:32:22.206181 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:22.206311 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":351,"threads_started":1}
I20260620 23:32:22.277356 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:22.276288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:35935)
0620 23:32:22.277052 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:22.277167 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":304,"threads_started":1}
I20260620 23:32:22.740664 15904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:22.739667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:53869)
0620 23:32:22.740306 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:22.740426 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":330,"threads_started":1}
I20260620 23:32:23.209101 15905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:23.207881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:54463)
0620 23:32:23.208621 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:23.208767 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":384,"threads_started":1}
I20260620 23:32:23.279531 15906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:23.278669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:42003)
0620 23:32:23.279235 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:23.279352 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":296,"threads_started":1}
I20260620 23:32:23.743105 15907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:23.742154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:39045)
0620 23:32:23.742788 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:23.742919 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
I20260620 23:32:24.211126 15908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:24.210271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:59387)
0620 23:32:24.210816 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:24.210919 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":264,"threads_started":1}
I20260620 23:32:24.281961 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:24.281081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:56551)
0620 23:32:24.281657 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:24.281773 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":309,"threads_started":1}
I20260620 23:32:24.745378 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:24.744381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:58681)
0620 23:32:24.744972 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:24.745127 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":280,"threads_started":1}
I20260620 23:32:25.213832 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:25.212818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:46789)
0620 23:32:25.213498 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:25.213620 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":367,"threads_started":1}
I20260620 23:32:25.284211 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:25.283287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:47455)
0620 23:32:25.283897 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:25.284005 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":323,"threads_started":1}
I20260620 23:32:25.747977 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:25.747084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.194:53639)
0620 23:32:25.747673 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:25.747789 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":285,"threads_started":1}
I20260620 23:32:26.216538 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:26.215497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.193:42433)
0620 23:32:26.216232 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:26.216348 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":371,"threads_started":1}
I20260620 23:32:26.253010 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 15431
I20260620 23:32:26.279753 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:26.254573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59538 (local address 127.0.96.193:42903)
0620 23:32:26.255292 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:26.255299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:26.255329 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:32:26.255434 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:26.255443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:26.255760 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:26.256698 (+ 938us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:26.258398 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:26.259418 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:26.260956 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:26.261821 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:26.261944 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:26.271945 (+ 10001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:26.271967 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:26.271985 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:26.272036 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:26.275639 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:26.276973 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:26.277032 (+ 59us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:26.277042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:26.277239 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:26.277772 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:26.277776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:26.277780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:26.278282 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:26.278618 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:26.279026 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 23:32:26.279347 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":381,"threads_started":1}
I20260620 23:32:26.281787 15590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:26.287362 15917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:26.286041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:35269 (local address 127.0.96.195:40053)
0620 23:32:26.286855 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:26.287071 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:35269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":384,"threads_started":1}
I20260620 23:32:26.384544 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 15431
I20260620 23:32:26.406111 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 15599
I20260620 23:32:26.421945 15918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:26.407023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.0.96.194:34257)
0620 23:32:26.407529 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:26.407534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:26.407554 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:32:26.407620 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:26.407623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:26.407705 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:26.407807 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:26.409658 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:26.410449 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:26.411940 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:26.412371 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:26.412475 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:26.415508 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:26.415513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:26.415518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:26.415553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:26.418592 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:26.419717 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:26.419739 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:26.419742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:26.419898 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:26.420316 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:26.420318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:26.420321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:26.420609 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:26.420867 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:26.421408 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 23:32:26.421603 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":271,"threads_started":1}
I20260620 23:32:26.422782 15722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:26.514635 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 15599
I20260620 23:32:26.534034 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 15730
I20260620 23:32:26.551152 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:26.535319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.96.195:38007)
0620 23:32:26.535966 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:26.535972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:26.535989 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:32:26.536046 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:26.536049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:26.536123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:26.536219 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:26.538188 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:26.539056 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:26.540582 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:26.541193 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:26.541322 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:26.544503 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:26.544509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:26.544513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:26.544554 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:26.547707 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:26.548970 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:26.548975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:26.548978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:26.549228 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:26.549648 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:26.549652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:26.549654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:26.549948 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:26.550167 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:26.550619 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 23:32:26.550813 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":334,"threads_started":1}
I20260620 23:32:26.552417 15854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:26.640955 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 15730
2026-06-20T23:32:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25561 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:32:26 dist-test-slave-v34r krb5kdc[15924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:32:26 dist-test-slave-v34r krb5kdc[15924](info): set up 2 sockets
Jun 20 23:32:26 dist-test-slave-v34r krb5kdc[15924](info): commencing operation
krb5kdc: starting...
W20260620 23:32:27.833750 387 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.010s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:32:28 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:32:28.168766 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36397
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:28.587023 15937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:28.587378 15937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:28.587523 15937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:28.596450 15937 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:32:28.596601 15937 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:32:28.596642 15937 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:32:28.596693 15937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:32:28.596843 15937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:28.596975 15937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:28.597107 15937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:32:28.597203 15937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:32:28.608214 15937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:28.609757 15937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:28.611157 15937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:28.625092 15945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:28.625247 15943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:28.625349 15937 server_base.cc:1034] running on GCE node
W20260620 23:32:28.625223 15942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:28.627250 15937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:28.629482 15937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:28.630890 15937 hybrid_clock.cc:648] HybridClock initialized: now 1781998348630769 us; error 94 us; skew 500 ppm
I20260620 23:32:28.631423 15937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:28 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:28.635921 15937 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:32:28.638371 15937 webserver.cc:460] Webserver started at http://127.0.96.254:43799/ using document root <none> and password file <none>
I20260620 23:32:28.639429 15937 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:28.639607 15937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:28.640012 15937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:28.644140 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "e2c547b6250e4f3ebc29b3f97a426945"
format_stamp: "Formatted at 2026-06-20 23:32:28 on dist-test-slave-v34r"
server_key: "daedfc3c751a8cce5f3f8e711ee3ab80"
server_key_iv: "932f7d56e402e4e85ca1404f596dfcaa"
server_key_version: "encryptionkey@0"
I20260620 23:32:28.645349 15937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "e2c547b6250e4f3ebc29b3f97a426945"
format_stamp: "Formatted at 2026-06-20 23:32:28 on dist-test-slave-v34r"
server_key: "daedfc3c751a8cce5f3f8e711ee3ab80"
server_key_iv: "932f7d56e402e4e85ca1404f596dfcaa"
server_key_version: "encryptionkey@0"
I20260620 23:32:28.653692 15937 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260620 23:32:28.659003 15952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:28.661324 15937 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:32:28.661636 15937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "e2c547b6250e4f3ebc29b3f97a426945"
format_stamp: "Formatted at 2026-06-20 23:32:28 on dist-test-slave-v34r"
server_key: "daedfc3c751a8cce5f3f8e711ee3ab80"
server_key_iv: "932f7d56e402e4e85ca1404f596dfcaa"
server_key_version: "encryptionkey@0"
I20260620 23:32:28.661991 15937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:28.690091 15937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:28.696043 15937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:28.696491 15937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:28.721349 15937 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36397
I20260620 23:32:28.721405 16003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36397 every 8 connection(s)
I20260620 23:32:28.723258 15937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:32:28.725049 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 15937
I20260620 23:32:28.725435 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:32:28.726377 387 external_mini_cluster.cc:1428] Setting key f0c7d6165f30a6e47515a45b34c981aa
I20260620 23:32:28.729900 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:28.744670 16004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap starting.
Jun 20 23:32:28 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:28.748577 16004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:28.750070 16004 log.cc:824] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:28.753945 16004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: No bootstrap required, opened a new log
I20260620 23:32:28.756954 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:28.729734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.0.96.254:36397)
0620 23:32:28.730850 (+ 1116us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:28.730867 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:28.730934 (+ 67us) server_negotiation.cc:407] Connection header received
0620 23:32:28.733419 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:28.733451 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:28.734033 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:28.734543 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:28.736671 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:28.738840 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:28.740412 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:28.741623 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:28.741844 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:28.747271 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:28.747283 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:28.747310 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:28.747371 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:28.751661 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:28.752735 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:28.752745 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:28.752753 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:28.753022 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:28.753527 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:28.753532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:28.753535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:28.754252 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:28.754601 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:28.755053 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 23:32:28.755524 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":524,"threads_started":1}
I20260620 23:32:28.763523 16004 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:28.763913 16004 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:28.763983 16004 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c547b6250e4f3ebc29b3f97a426945, State: Initialized, Role: FOLLOWER
I20260620 23:32:28.764580 16004 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:28.764799 16004 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:32:28.764892 16004 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:32:28.765112 16004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:28.767616 16004 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:28.768271 16004 leader_election.cc:302] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c547b6250e4f3ebc29b3f97a426945; no voters:
I20260620 23:32:28.769258 16004 leader_election.cc:288] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:32:28.769443 16009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:28.770435 16009 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 LEADER]: Becoming Leader. State: Replica: e2c547b6250e4f3ebc29b3f97a426945, State: Running, Role: LEADER
I20260620 23:32:28.771636 16004 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:32:28.771502 16009 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:28.775179 16011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c547b6250e4f3ebc29b3f97a426945. Latest consensus state: current_term: 1 leader_uuid: "e2c547b6250e4f3ebc29b3f97a426945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } } }
I20260620 23:32:28.775175 16010 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2c547b6250e4f3ebc29b3f97a426945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } } }
I20260620 23:32:28.776013 16011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:28.776050 16010 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:28.779470 16016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:32:28.786304 16016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:32:28.798105 16016 catalog_manager.cc:1347] Generated new cluster ID: a9e2a91f5da241ba8f2e9e9de78477cc
I20260620 23:32:28.798302 16016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:32:28.816247 16016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:32:28.818434 16016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:32:28.826359 16016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Generated new TSK 0
I20260620 23:32:28.827490 16016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:32:28.906605 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36397
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:29.338999 16030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:29.339447 16030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:29.339635 16030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:29.348057 16030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:29.348284 16030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:29.348515 16030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:32:29.360759 16030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:29.362540 16030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:29.364187 16030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:29.381662 16036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:29.381670 16035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:29.382072 16038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:29.382854 16030 server_base.cc:1034] running on GCE node
I20260620 23:32:29.383598 16030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:29.384689 16030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:29.385998 16030 hybrid_clock.cc:648] HybridClock initialized: now 1781998349385922 us; error 70 us; skew 500 ppm
I20260620 23:32:29.386464 16030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:29 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:29.390053 16030 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:32:29.391796 16030 webserver.cc:460] Webserver started at http://127.0.96.193:42123/ using document root <none> and password file <none>
I20260620 23:32:29.392606 16030 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:29.392760 16030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:29.393160 16030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:29.396302 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "c986ab972558408b871105ebe5bea8bb"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "7e603246419aae758fa8ddcd9c874ca4"
server_key_iv: "a68d695b3c61f68fd050d887f1593994"
server_key_version: "encryptionkey@0"
I20260620 23:32:29.397188 16030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "c986ab972558408b871105ebe5bea8bb"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "7e603246419aae758fa8ddcd9c874ca4"
server_key_iv: "a68d695b3c61f68fd050d887f1593994"
server_key_version: "encryptionkey@0"
I20260620 23:32:29.403321 16030 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:32:29.408434 16045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:29.410465 16030 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:32:29.410758 16030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "c986ab972558408b871105ebe5bea8bb"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "7e603246419aae758fa8ddcd9c874ca4"
server_key_iv: "a68d695b3c61f68fd050d887f1593994"
server_key_version: "encryptionkey@0"
I20260620 23:32:29.411003 16030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:29.429589 16030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:29.435294 16030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:29.435693 16030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:29.437078 16030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:29.438836 16030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:29.438937 16030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:29.439098 16030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:29.439167 16030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:29.480454 16030 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:32871
I20260620 23:32:29.480571 16157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:32871 every 8 connection(s)
I20260620 23:32:29.482342 16030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:32:29.492542 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16030
I20260620 23:32:29.493149 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:32:29.494004 387 external_mini_cluster.cc:1428] Setting key 544a186c6bb0845fa582f7e7b6ad668e
Jun 20 23:32:29 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:29.508708 16160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:29.485638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:60605 (local address 127.0.96.254:36397)
0620 23:32:29.486265 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:29.486280 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:29.487748 (+ 1468us) server_negotiation.cc:407] Connection header received
0620 23:32:29.489923 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:29.489929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:29.490034 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:29.490161 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:29.491900 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:29.492647 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:29.494529 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:29.494959 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:29.495149 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:29.500599 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:29.500613 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:29.500617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:29.500666 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:29.504154 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:29.505491 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:29.505498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:29.505500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:29.505682 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:29.506284 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:29.506295 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:29.506297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:29.506692 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:29.506894 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:29.507898 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0620 23:32:29.508235 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":321,"threads_started":1}
I20260620 23:32:29.510221 16161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:29.486408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:60605)
0620 23:32:29.487515 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:29.487573 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:29.489612 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:29.490442 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:29.490452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:29.491071 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:29.491711 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:29.491730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:29.492830 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:29.492834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:29.494362 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:29.494377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:29.495377 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:29.495387 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:29.495846 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:29.496929 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:29.496968 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:29.500215 (+ 3247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:29.504468 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:29.504494 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:29.504516 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:29.505286 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:29.505835 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:29.505841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:29.505847 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:29.506125 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:29.506855 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:29.506866 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:29.507396 (+ 530us) client_negotiation.cc:769] Sending connection context
0620 23:32:29.507761 (+ 365us) client_negotiation.cc:240] Negotiation successful
0620 23:32:29.508493 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":450,"threads_started":1}
I20260620 23:32:29.512584 16158 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36397
I20260620 23:32:29.513088 16158 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:29.514194 16158 heartbeater.cc:511] Master 127.0.96.254:36397 requested a full tablet report, sending...
I20260620 23:32:29.517369 15969 ts_manager.cc:194] Registered new tserver with Master: c986ab972558408b871105ebe5bea8bb (127.0.96.193:32871)
I20260620 23:32:29.519734 15969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:60605
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:32:29.557163 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36397
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:29.915126 16164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:29.915436 16164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:29.915572 16164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:29.924113 16164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:29.924268 16164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:29.924466 16164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:32:29.935398 16164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:29.936892 16164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:29.938326 16164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:29.953922 16172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:29.953981 16170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:29.953868 16169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:29.955937 16164 server_base.cc:1034] running on GCE node
I20260620 23:32:29.956741 16164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:29.958166 16164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:29.959473 16164 hybrid_clock.cc:648] HybridClock initialized: now 1781998349959361 us; error 116 us; skew 500 ppm
I20260620 23:32:29.959885 16164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:29 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:29.964043 16164 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:32:29.966100 16164 webserver.cc:460] Webserver started at http://127.0.96.194:44629/ using document root <none> and password file <none>
I20260620 23:32:29.966960 16164 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:29.967126 16164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:29.967496 16164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:29.970806 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "5d9f3d7b1f3a4781938e397efa70db26"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "cd0fded913c8adfbac585c1ddb3438f0"
server_key_iv: "27134c282b00bd8785452ebb063cac89"
server_key_version: "encryptionkey@0"
I20260620 23:32:29.971731 16164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "5d9f3d7b1f3a4781938e397efa70db26"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "cd0fded913c8adfbac585c1ddb3438f0"
server_key_iv: "27134c282b00bd8785452ebb063cac89"
server_key_version: "encryptionkey@0"
I20260620 23:32:29.978158 16164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 23:32:29.982983 16179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:29.984817 16164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260620 23:32:29.985105 16164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "5d9f3d7b1f3a4781938e397efa70db26"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "cd0fded913c8adfbac585c1ddb3438f0"
server_key_iv: "27134c282b00bd8785452ebb063cac89"
server_key_version: "encryptionkey@0"
I20260620 23:32:29.985335 16164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:30.013113 16164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:30.018106 16164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:30.018451 16164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:30.021718 16164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:30.024513 16164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:30.024632 16164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:30.024770 16164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:30.024832 16164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:30.067392 16164 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34011
I20260620 23:32:30.067427 16291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34011 every 8 connection(s)
I20260620 23:32:30.068813 16164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:32:30.078904 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16164
I20260620 23:32:30.079111 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:32:30.079713 387 external_mini_cluster.cc:1428] Setting key e725f4f339e287d186727637f11e12da
Jun 20 23:32:30 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:30.096648 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.072228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:33351 (local address 127.0.96.254:36397)
0620 23:32:30.072761 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.072767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.074142 (+ 1375us) server_negotiation.cc:407] Connection header received
0620 23:32:30.076421 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.076428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.076552 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.076694 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:30.078858 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.080002 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.082376 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.082886 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.083051 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.087968 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:30.087979 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:30.087988 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:30.088055 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:30.091621 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:30.093122 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:30.093128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:30.093130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:30.093336 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:30.094029 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:30.094033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:30.094035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:30.094424 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:30.094562 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.096020 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.096363 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":239,"threads_started":1}
I20260620 23:32:30.098600 16295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.072844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:33351)
0620 23:32:30.073877 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.073944 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.076037 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.077067 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.077079 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.077763 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:30.078630 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.078653 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.080192 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.080196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.082137 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.082167 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.083189 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.083197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.083484 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.084413 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:30.084444 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:30.087478 (+ 3034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:30.091853 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:30.091870 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:30.091897 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:30.092815 (+ 918us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:30.093487 (+ 672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:30.093496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:30.093503 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:30.093852 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:30.094596 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:30.094607 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:30.095307 (+ 700us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.095734 (+ 427us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.096549 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":417,"threads_started":1}
I20260620 23:32:30.100659 16292 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36397
I20260620 23:32:30.101147 16292 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:30.101985 16292 heartbeater.cc:511] Master 127.0.96.254:36397 requested a full tablet report, sending...
I20260620 23:32:30.103658 15969 ts_manager.cc:194] Registered new tserver with Master: 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011)
I20260620 23:32:30.104599 15969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:33351
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:32:30.148537 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36397
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:30.506182 16298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:30.506541 16298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:30.506711 16298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:30.514418 16298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:30.514600 16298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:30.514827 16298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:32:30.523934 16158 heartbeater.cc:503] Master 127.0.96.254:36397 was elected leader, sending a full tablet report...
I20260620 23:32:30.525552 16298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:30.527113 16298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:30.528422 16298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:30.543310 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:30.543394 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:30.544035 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:30.545223 16298 server_base.cc:1034] running on GCE node
I20260620 23:32:30.545964 16298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:30.547130 16298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:30.548398 16298 hybrid_clock.cc:648] HybridClock initialized: now 1781998350548326 us; error 83 us; skew 500 ppm
I20260620 23:32:30.548758 16298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:30 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:30.552157 16298 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:32:30.554055 16298 webserver.cc:460] Webserver started at http://127.0.96.195:35333/ using document root <none> and password file <none>
I20260620 23:32:30.554723 16298 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:30.554827 16298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:30.555094 16298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:32:30.558139 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "100be9b823c5407bb6999f7850efc749"
format_stamp: "Formatted at 2026-06-20 23:32:30 on dist-test-slave-v34r"
server_key: "32319f2060584ad883909f95bf6064dd"
server_key_iv: "99a1471214a7e4989f5a7c5a663180d5"
server_key_version: "encryptionkey@0"
I20260620 23:32:30.558846 16298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "100be9b823c5407bb6999f7850efc749"
format_stamp: "Formatted at 2026-06-20 23:32:30 on dist-test-slave-v34r"
server_key: "32319f2060584ad883909f95bf6064dd"
server_key_iv: "99a1471214a7e4989f5a7c5a663180d5"
server_key_version: "encryptionkey@0"
I20260620 23:32:30.564509 16298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 23:32:30.568759 16313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:30.570677 16298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:32:30.570992 16298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "100be9b823c5407bb6999f7850efc749"
format_stamp: "Formatted at 2026-06-20 23:32:30 on dist-test-slave-v34r"
server_key: "32319f2060584ad883909f95bf6064dd"
server_key_iv: "99a1471214a7e4989f5a7c5a663180d5"
server_key_version: "encryptionkey@0"
I20260620 23:32:30.571216 16298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:30.599887 16298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:30.605841 16298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:30.606170 16298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:30.607342 16298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:30.609083 16298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:30.609229 16298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 23:32:30.609607 16298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:30.609723 16298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:30.651962 16298 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:35013
I20260620 23:32:30.652065 16425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:35013 every 8 connection(s)
I20260620 23:32:30.653563 16298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:32:30.658355 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16298
I20260620 23:32:30.658625 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:32:30.659312 387 external_mini_cluster.cc:1428] Setting key 181bb50a4a7260f2a9bab5bf954a4ef7
Jun 20 23:32:30 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:30.682574 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.657167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:34393 (local address 127.0.96.254:36397)
0620 23:32:30.658003 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.658017 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.659486 (+ 1469us) server_negotiation.cc:407] Connection header received
0620 23:32:30.662539 (+ 3053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.662548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.662673 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.662818 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:30.665074 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.665880 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.668058 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.668490 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.668596 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.674067 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:30.674074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:30.674078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:30.674121 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:30.678297 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:30.679614 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:30.679619 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:30.679622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:30.679806 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:30.680359 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:30.680363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:30.680365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:30.680690 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:30.680864 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.682028 (+ 1164us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.682222 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":291,"threads_started":1}
I20260620 23:32:30.684003 16429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.657657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:34393)
0620 23:32:30.659125 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.659210 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.662132 (+ 2922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.663164 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.663181 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.664019 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:30.664813 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.664835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.666108 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.666113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.667881 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.667896 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.668805 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.668812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.669152 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.670133 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:30.670165 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:30.673625 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:30.678572 (+ 4947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:30.678593 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:30.678616 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:30.679434 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:30.679971 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:30.679975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:30.679978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:30.680228 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:30.680875 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:30.680884 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:30.681580 (+ 696us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.681929 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.682503 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":486,"threads_started":1}
I20260620 23:32:30.686198 16426 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36397
I20260620 23:32:30.686669 16426 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:30.687744 16426 heartbeater.cc:511] Master 127.0.96.254:36397 requested a full tablet report, sending...
I20260620 23:32:30.690069 15969 ts_manager.cc:194] Registered new tserver with Master: 100be9b823c5407bb6999f7850efc749 (127.0.96.195:35013)
I20260620 23:32:30.690796 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:32:30.691078 15969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:34393
I20260620 23:32:30.713546 16428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.697608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38250 (local address 127.0.96.254:36397)
0620 23:32:30.697824 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.697829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.698277 (+ 448us) server_negotiation.cc:407] Connection header received
0620 23:32:30.698527 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.698532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.698616 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.698749 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:30.700377 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.701332 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.702810 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.703311 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.703451 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.706410 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:30.706417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:30.706421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:30.706473 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:30.709690 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:30.710605 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:30.710611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:30.710614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:30.710802 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:30.711473 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:30.711478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:30.711481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:30.711943 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:30.712186 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.712830 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.713282 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:32:30.719074 15969 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:32:30.727813 15969 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:32:30.772665 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.744863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.0.96.193:32871)
0620 23:32:30.762353 (+ 17490us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.762367 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.762413 (+ 46us) server_negotiation.cc:407] Connection header received
0620 23:32:30.762564 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.762576 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.762961 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.763242 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:30.764386 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.766085 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.769255 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.771078 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.771264 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.771588 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.772000 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.772259 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17195,"thread_start_us":16975,"threads_started":1}
I20260620 23:32:30.772701 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.744436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:59042)
0620 23:32:30.759972 (+ 15536us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.760056 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.760375 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.763578 (+ 3203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.763587 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.763634 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:30.764118 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.764137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.766254 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.766266 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.768914 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.768928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.771359 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.771367 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.771710 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.771778 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.771929 (+ 151us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.772159 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15014,"mutex_wait_us":4,"thread_start_us":16237,"threads_started":1}
I20260620 23:32:30.773841 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.745452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.0.96.194:34011)
0620 23:32:30.755078 (+ 9626us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.755096 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.763929 (+ 8833us) server_negotiation.cc:407] Connection header received
0620 23:32:30.764913 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.764922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.765217 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.765435 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:30.766404 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.767929 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.770476 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.771754 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.771906 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.772125 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.773230 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.773472 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9298,"thread_start_us":9053,"threads_started":1}
I20260620 23:32:30.774829 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.744910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34011 (local address 127.0.0.1:46376)
0620 23:32:30.763672 (+ 18762us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.763710 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.764601 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.765452 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.765457 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.765484 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:30.766137 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.766150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.768107 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.768112 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.770309 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.770322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.772317 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.772324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.772534 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.772570 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.774112 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.774359 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18534,"mutex_wait_us":39,"thread_start_us":18317,"threads_started":1}
I20260620 23:32:30.775458 16441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.745434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:35013 (local address 127.0.0.1:57944)
0620 23:32:30.764466 (+ 19032us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.764485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.764607 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.765741 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.765747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.765778 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:30.766553 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.766569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.769303 (+ 2734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.769315 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.771692 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.771706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.774506 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.774511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.774706 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.774740 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.774857 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.775041 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18916,"mutex_wait_us":82,"thread_start_us":18750,"threads_started":1}
I20260620 23:32:30.776701 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.746237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.0.96.195:35013)
0620 23:32:30.762971 (+ 16734us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.762986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.764747 (+ 1761us) server_negotiation.cc:407] Connection header received
0620 23:32:30.764879 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.764886 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.765236 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.765478 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:30.766869 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.769134 (+ 2265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.771915 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.773933 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.774078 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.774347 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.775910 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.776292 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16467,"thread_start_us":16231,"threads_started":1}
I20260620 23:32:30.778522 16227 tablet_service.cc:1467] Processing CreateTablet for tablet ad5fcf15641345a2837ad3bdb909bc2f (DEFAULT_TABLE table=test-table [id=6d1f628bfe3049c98e6638820a6b6623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:30.778568 16093 tablet_service.cc:1467] Processing CreateTablet for tablet ad5fcf15641345a2837ad3bdb909bc2f (DEFAULT_TABLE table=test-table [id=6d1f628bfe3049c98e6638820a6b6623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:30.780030 16227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5fcf15641345a2837ad3bdb909bc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:30.780436 16093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5fcf15641345a2837ad3bdb909bc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:30.781100 16361 tablet_service.cc:1467] Processing CreateTablet for tablet ad5fcf15641345a2837ad3bdb909bc2f (DEFAULT_TABLE table=test-table [id=6d1f628bfe3049c98e6638820a6b6623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:30.783107 16361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad5fcf15641345a2837ad3bdb909bc2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:30.797797 16445 tablet_bootstrap.cc:492] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Bootstrap starting.
I20260620 23:32:30.800289 16446 tablet_bootstrap.cc:492] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Bootstrap starting.
I20260620 23:32:30.802163 16447 tablet_bootstrap.cc:492] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Bootstrap starting.
I20260620 23:32:30.802548 16445 tablet_bootstrap.cc:654] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:30.804234 16445 log.cc:824] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:30.806006 16447 tablet_bootstrap.cc:654] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:30.807394 16447 log.cc:824] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:30.808409 16445 tablet_bootstrap.cc:492] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: No bootstrap required, opened a new log
I20260620 23:32:30.808559 16446 tablet_bootstrap.cc:654] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:30.808738 16445 ts_tablet_manager.cc:1397] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:32:30.810086 16446 log.cc:824] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:30.811086 16447 tablet_bootstrap.cc:492] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: No bootstrap required, opened a new log
I20260620 23:32:30.811391 16447 ts_tablet_manager.cc:1397] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 23:32:30.814164 16446 tablet_bootstrap.cc:492] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: No bootstrap required, opened a new log
I20260620 23:32:30.814620 16446 ts_tablet_manager.cc:1397] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260620 23:32:30.819803 16445 raft_consensus.cc:348] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.820319 16445 raft_consensus.cc:374] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:30.820423 16445 raft_consensus.cc:729] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9f3d7b1f3a4781938e397efa70db26, State: Initialized, Role: FOLLOWER
I20260620 23:32:30.821154 16445 consensus_queue.cc:260] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.822299 16447 raft_consensus.cc:348] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.822664 16292 heartbeater.cc:503] Master 127.0.96.254:36397 was elected leader, sending a full tablet report...
I20260620 23:32:30.822737 16447 raft_consensus.cc:374] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:30.822835 16447 raft_consensus.cc:729] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100be9b823c5407bb6999f7850efc749, State: Initialized, Role: FOLLOWER
I20260620 23:32:30.823258 16445 ts_tablet_manager.cc:1428] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260620 23:32:30.823553 16447 consensus_queue.cc:260] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.824913 16426 heartbeater.cc:503] Master 127.0.96.254:36397 was elected leader, sending a full tablet report...
I20260620 23:32:30.825425 16447 ts_tablet_manager.cc:1428] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260620 23:32:30.825932 16446 raft_consensus.cc:348] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.826478 16446 raft_consensus.cc:374] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:30.826586 16446 raft_consensus.cc:729] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c986ab972558408b871105ebe5bea8bb, State: Initialized, Role: FOLLOWER
I20260620 23:32:30.827446 16446 consensus_queue.cc:260] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.829648 16446 ts_tablet_manager.cc:1428] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 23:32:30.905673 16427 tablet.cc:2369] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:30.957316 16453 raft_consensus.cc:482] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:30.957782 16453 raft_consensus.cc:504] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.960429 16453 leader_election.cc:288] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011), 100be9b823c5407bb6999f7850efc749 (127.0.96.195:35013)
I20260620 23:32:30.968657 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.961127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:37779 (local address 127.0.96.194:34011)
0620 23:32:30.961378 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.961384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.962403 (+ 1019us) server_negotiation.cc:407] Connection header received
0620 23:32:30.962655 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.962665 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.962727 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.962836 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:30.963623 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.964326 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.966489 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.967363 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.967620 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.967717 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.968147 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.968285 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:32:30.968781 16454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.960918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34011 (local address 127.0.96.193:37779)
0620 23:32:30.962261 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.962282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.962494 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.962873 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.962879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.962929 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:30.963437 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.963449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.964493 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.964499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.966292 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.966313 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.967701 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.967709 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.967928 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.967978 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.968097 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.968284 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1210,"mutex_wait_us":4,"thread_start_us":584,"threads_started":1}
I20260620 23:32:30.969099 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.961207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:54325 (local address 127.0.96.195:35013)
0620 23:32:30.961631 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:30.961638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:30.962257 (+ 619us) server_negotiation.cc:407] Connection header received
0620 23:32:30.962537 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:30.962542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:30.962620 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:30.962733 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:30.963431 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.964396 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.966743 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.967838 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.968024 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.968127 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:30.968605 (+ 478us) server_negotiation.cc:299] Negotiation successful
0620 23:32:30.968789 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260620 23:32:30.969589 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:30.960868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:35013 (local address 127.0.96.193:54325)
0620 23:32:30.962095 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:30.962125 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:30.962357 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:30.962755 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:30.962760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:30.962789 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:30.963257 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.963267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.964567 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.964576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.966520 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:30.966542 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:30.968170 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:30.968178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:30.968375 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:30.968420 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:32:30.968538 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:32:30.968715 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":4,"thread_start_us":1040,"threads_started":1}
I20260620 23:32:30.970494 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5fcf15641345a2837ad3bdb909bc2f" candidate_uuid: "c986ab972558408b871105ebe5bea8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9f3d7b1f3a4781938e397efa70db26" is_pre_election: true
I20260620 23:32:30.970996 16381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5fcf15641345a2837ad3bdb909bc2f" candidate_uuid: "c986ab972558408b871105ebe5bea8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100be9b823c5407bb6999f7850efc749" is_pre_election: true
I20260620 23:32:30.971192 16247 raft_consensus.cc:2454] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c986ab972558408b871105ebe5bea8bb in term 0.
I20260620 23:32:30.971634 16381 raft_consensus.cc:2454] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c986ab972558408b871105ebe5bea8bb in term 0.
I20260620 23:32:30.972299 16048 leader_election.cc:302] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d9f3d7b1f3a4781938e397efa70db26, c986ab972558408b871105ebe5bea8bb; no voters:
I20260620 23:32:30.972764 16453 raft_consensus.cc:2781] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:32:30.972944 16453 raft_consensus.cc:482] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:32:30.973079 16453 raft_consensus.cc:3037] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:30.975534 16453 raft_consensus.cc:504] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
I20260620 23:32:30.976727 16453 leader_election.cc:288] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [CANDIDATE]: Term 1 election: Requested vote from peers 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011), 100be9b823c5407bb6999f7850efc749 (127.0.96.195:35013)
I20260620 23:32:30.977509 16381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5fcf15641345a2837ad3bdb909bc2f" candidate_uuid: "c986ab972558408b871105ebe5bea8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100be9b823c5407bb6999f7850efc749"
I20260620 23:32:30.977756 16381 raft_consensus.cc:3037] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:30.979357 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad5fcf15641345a2837ad3bdb909bc2f" candidate_uuid: "c986ab972558408b871105ebe5bea8bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9f3d7b1f3a4781938e397efa70db26"
I20260620 23:32:30.979593 16247 raft_consensus.cc:3037] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:30.980858 16381 raft_consensus.cc:2454] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c986ab972558408b871105ebe5bea8bb in term 1.
I20260620 23:32:30.981637 16049 leader_election.cc:302] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100be9b823c5407bb6999f7850efc749, c986ab972558408b871105ebe5bea8bb; no voters:
I20260620 23:32:30.982075 16453 raft_consensus.cc:2781] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:30.982375 16247 raft_consensus.cc:2454] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c986ab972558408b871105ebe5bea8bb in term 1.
I20260620 23:32:30.982740 16453 raft_consensus.cc:686] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 1 LEADER]: Becoming Leader. State: Replica: c986ab972558408b871105ebe5bea8bb, State: Running, Role: LEADER
I20260620 23:32:30.983412 16453 consensus_queue.cc:237] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } }
W20260620 23:32:30.985445 16159 tablet.cc:2369] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:30.989650 15969 catalog_manager.cc:5495] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c986ab972558408b871105ebe5bea8bb (127.0.96.193). New cstate: current_term: 1 leader_uuid: "c986ab972558408b871105ebe5bea8bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } health_report { overall_health: UNKNOWN } } }
I20260620 23:32:31.021286 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:31.014933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.96.193:32871)
0620 23:32:31.015164 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:31.015169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:31.015191 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:32:31.015322 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:31.015327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:31.015447 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:31.015640 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:32:31.016221 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.017138 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.018702 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.019205 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.019337 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:31.019876 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:32:31.020553 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:32:31.020696 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:31.020856 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 23:32:31.020948 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 23:32:31.031028 16247 raft_consensus.cc:1261] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Refusing update from remote peer c986ab972558408b871105ebe5bea8bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:31.031376 16381 raft_consensus.cc:1261] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Refusing update from remote peer c986ab972558408b871105ebe5bea8bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:31.032114 16456 consensus_queue.cc:1035] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:31.032553 16453 consensus_queue.cc:1035] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:31.047174 16459 mvcc.cc:205] Tried to move back new op lower bound from 7299065245812125696 to 7299065245632970752. Current Snapshot: MvccSnapshot[applied={T|T < 7299065245812125696}]
I20260620 23:32:31.064841 15969 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:31.065181 15969 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:31.069854 15969 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Sending DeleteTablet for 3 replicas of tablet ad5fcf15641345a2837ad3bdb909bc2f
I20260620 23:32:31.071457 16093 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5fcf15641345a2837ad3bdb909bc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:31 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:59042
I20260620 23:32:31.071669 16361 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5fcf15641345a2837ad3bdb909bc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:31 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:57944
I20260620 23:32:31.073074 16227 tablet_service.cc:1514] Processing DeleteTablet for tablet ad5fcf15641345a2837ad3bdb909bc2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:31 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:46376
I20260620 23:32:31.074368 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16030
I20260620 23:32:31.092271 16468 tablet_replica.cc:330] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: stopping tablet replica
I20260620 23:32:31.095851 16468 raft_consensus.cc:2229] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 1 LEADER]: Raft consensus shutting down.
Jun 20 23:32:31 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:32:31.101392 16470 tablet_replica.cc:330] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: stopping tablet replica
I20260620 23:32:31.101926 16470 raft_consensus.cc:2229] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:31.101925 16468 raft_consensus.cc:2258] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:31.102501 16470 raft_consensus.cc:2258] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:31.103400 16469 tablet_replica.cc:330] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: stopping tablet replica
I20260620 23:32:31.103833 16469 raft_consensus.cc:2229] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:31.104344 16469 raft_consensus.cc:2258] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:31.106436 16469 ts_tablet_manager.cc:1905] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:31.107841 16470 ts_tablet_manager.cc:1905] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:31.106535 16468 ts_tablet_manager.cc:1905] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:31.113118 16469 ts_tablet_manager.cc:1918] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:31.113309 16469 log.cc:1196] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/ad5fcf15641345a2837ad3bdb909bc2f
I20260620 23:32:31.113842 16469 ts_tablet_manager.cc:1939] T ad5fcf15641345a2837ad3bdb909bc2f P 100be9b823c5407bb6999f7850efc749: Deleting consensus metadata
I20260620 23:32:31.116003 15956 catalog_manager.cc:4841] TS 100be9b823c5407bb6999f7850efc749 (127.0.96.195:35013): tablet ad5fcf15641345a2837ad3bdb909bc2f (table test-table [id=6d1f628bfe3049c98e6638820a6b6623]) successfully deleted
I20260620 23:32:31.117197 16468 ts_tablet_manager.cc:1918] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:31.117413 16468 log.cc:1196] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/ad5fcf15641345a2837ad3bdb909bc2f
I20260620 23:32:31.117610 16470 ts_tablet_manager.cc:1918] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:31.117923 16470 log.cc:1196] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/ad5fcf15641345a2837ad3bdb909bc2f
I20260620 23:32:31.118001 16468 ts_tablet_manager.cc:1939] T ad5fcf15641345a2837ad3bdb909bc2f P c986ab972558408b871105ebe5bea8bb: Deleting consensus metadata
I20260620 23:32:31.118702 16470 ts_tablet_manager.cc:1939] T ad5fcf15641345a2837ad3bdb909bc2f P 5d9f3d7b1f3a4781938e397efa70db26: Deleting consensus metadata
I20260620 23:32:31.120039 15954 catalog_manager.cc:4841] TS c986ab972558408b871105ebe5bea8bb (127.0.96.193:32871): tablet ad5fcf15641345a2837ad3bdb909bc2f (table test-table [id=6d1f628bfe3049c98e6638820a6b6623]) successfully deleted
I20260620 23:32:31.120862 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:31.075466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.0.96.193:32871)
0620 23:32:31.075720 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:31.075727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:31.080071 (+ 4344us) server_negotiation.cc:407] Connection header received
0620 23:32:31.080309 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:31.080317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:31.080450 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:31.080635 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:31.085634 (+ 4999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.086966 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.092781 (+ 5815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.093678 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.093835 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:31.105406 (+ 11571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:31.105429 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:31.105450 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:31.105513 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:31.110571 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:31.117219 (+ 6648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:31.117233 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:31.117237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:31.117571 (+ 334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:31.118216 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:31.118227 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:31.118231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:31.118913 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:31.119305 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:31.120075 (+ 770us) server_negotiation.cc:299] Negotiation successful
0620 23:32:31.120433 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:32:31.121547 15955 catalog_manager.cc:4841] TS 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011): tablet ad5fcf15641345a2837ad3bdb909bc2f (table test-table [id=6d1f628bfe3049c98e6638820a6b6623]) successfully deleted
I20260620 23:32:31.122648 16153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:31.234469 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16030
I20260620 23:32:31.254734 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16164
Jun 20 23:32:31 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:32:31.281092 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:31.255617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.0.96.194:34011)
0620 23:32:31.255945 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:31.255953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:31.255995 (+ 42us) server_negotiation.cc:407] Connection header received
0620 23:32:31.256242 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:31.256247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:31.256332 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:31.256455 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:31.258327 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.259158 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.261116 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.261876 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.262099 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:31.267907 (+ 5808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:31.267930 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:31.267953 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:31.268013 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:31.276957 (+ 8944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:31.278175 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:31.278185 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:31.278187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:31.278382 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:31.278958 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:31.278966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:31.278973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:31.279481 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:31.279774 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:31.280179 (+ 405us) server_negotiation.cc:299] Negotiation successful
0620 23:32:31.280423 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 23:32:31.281888 16287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:31.385560 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16164
I20260620 23:32:31.410005 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16298
Jun 20 23:32:31 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:32:31.428198 16444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:31.410949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.0.96.195:35013)
0620 23:32:31.411138 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:31.411143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:31.411167 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:32:31.411346 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:31.411350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:31.411438 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:31.411549 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:31.413461 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.414205 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.415903 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:31.416286 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:31.416399 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:31.421594 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:31.421609 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:31.421626 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:31.421663 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:31.424598 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:31.425837 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:31.425844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:31.425847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:31.426011 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:31.426553 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:31.426557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:31.426559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:31.426942 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:31.427159 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:31.427677 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 23:32:31.427884 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:32:31.429143 16421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:31.521540 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16298
I20260620 23:32:31.539253 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 15937
I20260620 23:32:31.540647 15999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:31.653214 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 15937
I20260620 23:32:51.681267 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36397
--webserver_interface=127.0.96.254
--webserver_port=43799
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:32:52.110246 16476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:52.110671 16476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:52.110859 16476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:52.119985 16476 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:32:52.120163 16476 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:32:52.120208 16476 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:32:52.120332 16476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:32:52.120431 16476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:52.120554 16476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:52.120662 16476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:32:52.120757 16476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:32:52.133253 16476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=43799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:52.135160 16476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:52.136973 16476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:52.167243 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:52.167233 16482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:52.167233 16483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:52.168625 16476 server_base.cc:1034] running on GCE node
I20260620 23:32:52.169914 16476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:52.172192 16476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:52.173478 16476 hybrid_clock.cc:648] HybridClock initialized: now 1781998372173371 us; error 70 us; skew 500 ppm
I20260620 23:32:52.173976 16476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:52 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:52.178727 16476 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:32:52.181182 16476 webserver.cc:460] Webserver started at http://127.0.96.254:43799/ using document root <none> and password file <none>
I20260620 23:32:52.182273 16476 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:52.182449 16476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:52.192672 16476 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.002s
I20260620 23:32:52.197321 16492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:52.199714 16476 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:32:52.200055 16476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "e2c547b6250e4f3ebc29b3f97a426945"
format_stamp: "Formatted at 2026-06-20 23:32:28 on dist-test-slave-v34r"
server_key: "daedfc3c751a8cce5f3f8e711ee3ab80"
server_key_iv: "932f7d56e402e4e85ca1404f596dfcaa"
server_key_version: "encryptionkey@0"
I20260620 23:32:52.201637 16476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:52.238656 16476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:52.244894 16476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:52.245409 16476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:52.271138 16476 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36397
I20260620 23:32:52.271209 16543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36397 every 8 connection(s)
I20260620 23:32:52.273366 16476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:32:52.275446 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16476
I20260620 23:32:52.276605 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:32871
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:36397
--webserver_port=42123
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:32:52.285161 16544 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:32:52.294919 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap starting.
I20260620 23:32:52.322613 16544 log.cc:824] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:52.339481 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:32:52.340250 16544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap complete.
I20260620 23:32:52.351662 16544 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:52.352840 16544 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c547b6250e4f3ebc29b3f97a426945, State: Initialized, Role: FOLLOWER
I20260620 23:32:52.353827 16544 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:52.354158 16544 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:32:52.354301 16544 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:32:52.354583 16544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 1 FOLLOWER]: Advancing to term 2
I20260620 23:32:52.359614 16544 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:52.360526 16544 leader_election.cc:302] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c547b6250e4f3ebc29b3f97a426945; no voters:
I20260620 23:32:52.361712 16544 leader_election.cc:288] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 23:32:52.361908 16548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 23:32:52.363412 16548 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 LEADER]: Becoming Leader. State: Replica: e2c547b6250e4f3ebc29b3f97a426945, State: Running, Role: LEADER
I20260620 23:32:52.364280 16548 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:32:52.364784 16544 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:32:52.367440 16550 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c547b6250e4f3ebc29b3f97a426945. Latest consensus state: current_term: 2 leader_uuid: "e2c547b6250e4f3ebc29b3f97a426945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } } }
I20260620 23:32:52.368041 16550 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:52.368929 16549 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e2c547b6250e4f3ebc29b3f97a426945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } } }
I20260620 23:32:52.369378 16549 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: This master's current role is: LEADER
I20260620 23:32:52.374671 16554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:32:52.387606 16554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:32:52.390534 16554 catalog_manager.cc:1259] Loaded cluster ID: a9e2a91f5da241ba8f2e9e9de78477cc
I20260620 23:32:52.390815 16554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:32:52.397382 16554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:32:52.400234 16554 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Loaded TSK: 0
I20260620 23:32:52.435911 16554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Generated new TSK 1
I20260620 23:32:52.436406 16554 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:32:52.726207 16546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:52.726657 16546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:52.726850 16546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:52.735496 16546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:52.735719 16546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:52.735936 16546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:32:52.747967 16546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:32871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=42123
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:52.749882 16546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:52.751825 16546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:52.771845 16571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:52.772274 16574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:52.772706 16572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:52.773746 16546 server_base.cc:1034] running on GCE node
I20260620 23:32:52.774682 16546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:52.776135 16546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:52.777407 16546 hybrid_clock.cc:648] HybridClock initialized: now 1781998372777362 us; error 76 us; skew 500 ppm
I20260620 23:32:52.777907 16546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:52 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:52.781951 16546 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:32:52.784132 16546 webserver.cc:460] Webserver started at http://127.0.96.193:42123/ using document root <none> and password file <none>
I20260620 23:32:52.785216 16546 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:52.785413 16546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:52.793279 16546 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:32:52.797765 16581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:52.799767 16546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:32:52.800076 16546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "c986ab972558408b871105ebe5bea8bb"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "7e603246419aae758fa8ddcd9c874ca4"
server_key_iv: "a68d695b3c61f68fd050d887f1593994"
server_key_version: "encryptionkey@0"
I20260620 23:32:52.801555 16546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:52.833554 16546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:52.839260 16546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:52.839655 16546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:52.841092 16546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:52.842904 16546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:52.843022 16546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:52.843171 16546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:52.843263 16546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:52.885432 16546 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:32871
I20260620 23:32:52.885499 16693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:32871 every 8 connection(s)
I20260620 23:32:52.887487 16546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:32:52.891218 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16546
I20260620 23:32:52.892022 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:34011
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:36397
--webserver_port=44629
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:32:52 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998372, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:52.923183 16697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:52.891669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:35823)
0620 23:32:52.892802 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:52.892872 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:52.895541 (+ 2669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:52.898781 (+ 3240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:52.898796 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:52.899718 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:52.900811 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:52.900842 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:52.903066 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:52.903076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:52.905564 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:52.905588 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:52.907015 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:52.907026 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:52.907432 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:52.908466 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:52.908508 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:52.911932 (+ 3424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:52.917720 (+ 5788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:52.917730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:52.917745 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:52.918316 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:52.919124 (+ 808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:52.919128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:52.919131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:52.919349 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:52.920659 (+ 1310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:52.920665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:52.921180 (+ 515us) client_negotiation.cc:769] Sending connection context
0620 23:32:52.921454 (+ 274us) client_negotiation.cc:240] Negotiation successful
0620 23:32:52.921904 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":465,"threads_started":1}
I20260620 23:32:52.924249 16696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:52.891316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:35823 (local address 127.0.96.254:36397)
0620 23:32:52.892123 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:52.892142 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:52.893108 (+ 966us) server_negotiation.cc:407] Connection header received
0620 23:32:52.896637 (+ 3529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:52.896664 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:52.897352 (+ 688us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:52.897841 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:52.901271 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:52.902781 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:52.905796 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:52.906606 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:52.906853 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:52.912287 (+ 5434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:52.912296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:52.912314 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:52.912369 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:52.917424 (+ 5055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:52.918576 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:52.918592 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:52.918604 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:52.918945 (+ 341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:52.919520 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:52.919529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:52.919535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:52.920486 (+ 951us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:52.920763 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:52.921842 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0620 23:32:52.922588 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":362,"threads_started":1}
I20260620 23:32:52.927947 16694 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36397
I20260620 23:32:52.928321 16694 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:52.929257 16694 heartbeater.cc:511] Master 127.0.96.254:36397 requested a full tablet report, sending...
I20260620 23:32:52.931885 16509 ts_manager.cc:194] Registered new tserver with Master: c986ab972558408b871105ebe5bea8bb (127.0.96.193:32871)
I20260620 23:32:52.933544 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:35823
W20260620 23:32:53.321830 16698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:53.322281 16698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:53.322508 16698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:53.331071 16698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:53.331283 16698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:53.331498 16698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:32:53.343222 16698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:34011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=44629
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:53.345170 16698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:53.346935 16698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:53.363539 16704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:53.363519 16703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:53.363657 16706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:53.367132 16698 server_base.cc:1034] running on GCE node
I20260620 23:32:53.368050 16698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:53.369447 16698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:53.370736 16698 hybrid_clock.cc:648] HybridClock initialized: now 1781998373370644 us; error 71 us; skew 500 ppm
I20260620 23:32:53.371218 16698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:53 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:53.375242 16698 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:32:53.377713 16698 webserver.cc:460] Webserver started at http://127.0.96.194:44629/ using document root <none> and password file <none>
I20260620 23:32:53.378860 16698 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:53.379060 16698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:53.386366 16698 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:32:53.390206 16713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:53.392194 16698 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:32:53.392484 16698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "5d9f3d7b1f3a4781938e397efa70db26"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "cd0fded913c8adfbac585c1ddb3438f0"
server_key_iv: "27134c282b00bd8785452ebb063cac89"
server_key_version: "encryptionkey@0"
I20260620 23:32:53.394019 16698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:53.414690 16698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:53.419889 16698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:53.420258 16698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:53.421653 16698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:53.423547 16698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:53.423656 16698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:53.423840 16698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:53.423921 16698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:53.465207 16698 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34011
I20260620 23:32:53.465327 16825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34011 every 8 connection(s)
I20260620 23:32:53.467170 16698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:32:53.467895 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16698
I20260620 23:32:53.468755 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:35013
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:36397
--webserver_port=35333
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 23:32:53 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:53.492522 16828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:53.470678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:58227 (local address 127.0.96.254:36397)
0620 23:32:53.471317 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:53.471327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:53.472627 (+ 1300us) server_negotiation.cc:407] Connection header received
0620 23:32:53.475873 (+ 3246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:53.475879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:53.475991 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:53.476142 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:53.478345 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:53.479270 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:53.480906 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:53.481463 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:53.481614 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:53.485988 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:53.485994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:53.485998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:53.486042 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:53.489107 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:53.489918 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:53.489923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:53.489929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:53.490061 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:53.490524 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:53.490529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:53.490537 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:53.490925 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:53.491101 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:53.491864 (+ 763us) server_negotiation.cc:299] Negotiation successful
0620 23:32:53.492125 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":300,"threads_started":1}
I20260620 23:32:53.493697 16829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:53.471331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:58227)
0620 23:32:53.472360 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:53.472440 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:53.475454 (+ 3014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:53.476479 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:53.476494 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:53.477420 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:53.478102 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:53.478119 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:53.479449 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:53.479460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:53.480744 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:53.480757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:53.481730 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:53.481740 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:53.482006 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:53.482915 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:53.482940 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:53.485673 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:53.489286 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:53.489296 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:53.489316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:53.489777 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:53.490187 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:53.490192 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:53.490196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:53.490390 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:53.491068 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:53.491078 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:53.491479 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 23:32:53.491776 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 23:32:53.492340 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":370,"threads_started":1}
I20260620 23:32:53.495985 16826 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36397
I20260620 23:32:53.496336 16826 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:53.497205 16826 heartbeater.cc:511] Master 127.0.96.254:36397 requested a full tablet report, sending...
I20260620 23:32:53.499051 16509 ts_manager.cc:194] Registered new tserver with Master: 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011)
I20260620 23:32:53.499923 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:58227
W20260620 23:32:53.871258 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:32:53.871621 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:32:53.871806 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:32:53.879513 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:32:53.879691 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:32:53.879925 16830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:32:53.890792 16830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:35013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=35333
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:32:53.892182 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:32:53.893664 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:32:53.909626 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:53.909720 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:32:53.909668 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:32:53.910723 16830 server_base.cc:1034] running on GCE node
I20260620 23:32:53.911441 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:32:53.912501 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:32:53.913722 16830 hybrid_clock.cc:648] HybridClock initialized: now 1781998373913666 us; error 55 us; skew 500 ppm
I20260620 23:32:53.914140 16830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:32:53 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:32:53.917629 16830 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:32:53.919370 16830 webserver.cc:460] Webserver started at http://127.0.96.195:35333/ using document root <none> and password file <none>
I20260620 23:32:53.920090 16830 fs_manager.cc:359] Metadata directory not provided
I20260620 23:32:53.920208 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:32:53.926564 16830 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 23:32:53.930240 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:53.932021 16830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:32:53.932248 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "100be9b823c5407bb6999f7850efc749"
format_stamp: "Formatted at 2026-06-20 23:32:30 on dist-test-slave-v34r"
server_key: "32319f2060584ad883909f95bf6064dd"
server_key_iv: "99a1471214a7e4989f5a7c5a663180d5"
server_key_version: "encryptionkey@0"
I20260620 23:32:53.933416 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:32:53.938222 16694 heartbeater.cc:503] Master 127.0.96.254:36397 was elected leader, sending a full tablet report...
I20260620 23:32:53.958393 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:32:53.963138 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:32:53.963371 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:32:53.964474 16830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:32:53.966181 16830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:32:53.966277 16830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:53.966405 16830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:32:53.966472 16830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:32:54.006801 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:35013
I20260620 23:32:54.007074 16957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:35013 every 8 connection(s)
I20260620 23:32:54.008658 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:32:54.013608 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 16830
Jun 20 23:32:54 dist-test-slave-v34r krb5kdc[15924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:32:54.035262 16962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.014793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51916 (local address 127.0.96.254:36397)
0620 23:32:54.015515 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.015525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.016190 (+ 665us) server_negotiation.cc:407] Connection header received
0620 23:32:54.016375 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.016381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.016477 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.016619 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:54.018786 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.020484 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.023298 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.024116 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.024248 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.028177 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:54.028183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:54.028189 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:54.028240 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:54.031552 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.032729 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.032733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.032739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.032885 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.033412 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.033420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.033426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.033826 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:54.034128 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.034578 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.034875 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":506,"thread_start_us":317,"threads_started":1}
I20260620 23:32:54.037046 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.012128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:46483 (local address 127.0.96.254:36397)
0620 23:32:54.012688 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.012696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.014355 (+ 1659us) server_negotiation.cc:407] Connection header received
0620 23:32:54.016782 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.016791 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.016885 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.017070 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:54.019006 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.020649 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.023403 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.024133 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.024245 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.028761 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:54.028767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:54.028771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:54.028845 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:54.033110 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.034109 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.034118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.034121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.034307 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.034877 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.034885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.034888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.035288 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:54.035496 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.036387 (+ 891us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.036620 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":189,"server-negotiator.queue_time_us":398,"thread_start_us":246,"threads_started":1}
I20260620 23:32:54.038084 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.012903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:46483)
0620 23:32:54.014125 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.014206 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:54.016417 (+ 2211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:54.017342 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:54.017351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:54.017901 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:32:54.018705 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.018734 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.020866 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.020873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.023179 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.023193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.024468 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.024477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.024767 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.025756 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:32:54.025790 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:32:54.028371 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:32:54.033275 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:54.033282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:54.033296 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:54.033943 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:54.034464 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:32:54.034468 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:32:54.034471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:32:54.034710 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:32:54.035445 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:32:54.035455 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:32:54.035912 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 23:32:54.036290 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 23:32:54.036824 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":404,"threads_started":1}
I20260620 23:32:54.039739 16958 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36397
I20260620 23:32:54.040071 16958 heartbeater.cc:464] Registering TS with master...
I20260620 23:32:54.040804 16958 heartbeater.cc:511] Master 127.0.96.254:36397 requested a full tablet report, sending...
I20260620 23:32:54.042526 16509 ts_manager.cc:194] Registered new tserver with Master: 100be9b823c5407bb6999f7850efc749 (127.0.96.195:35013)
I20260620 23:32:54.043298 16509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:46483
I20260620 23:32:54.051954 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:32:54.059594 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.053127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.0.96.254:36397)
0620 23:32:54.053337 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.053342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.053980 (+ 638us) server_negotiation.cc:407] Connection header received
0620 23:32:54.054204 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.054209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.054378 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.054524 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:32:54.054998 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.055718 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.057412 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.057868 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.057978 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.058615 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:32:54.058885 (+ 270us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 23:32:54.059308 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51928: token expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 23:32:54.059731 16960 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51928: token expired
W20260620 23:32:54.060631 16433 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.96.254:36397)
I20260620 23:32:54.074359 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.061471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.0.96.254:36397)
0620 23:32:54.061667 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.061675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.061696 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:32:54.061746 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.061751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.061814 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.061921 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:54.063336 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.064065 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.065640 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.066098 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.066227 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.069066 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:54.069071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:54.069075 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:54.069112 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:54.071617 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.072286 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.072293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.072297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.072429 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.072820 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.072825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.072829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.073226 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:54.073515 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.073810 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.074028 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:32:54.078426 16509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:32:54.085970 16509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:32:54.137122 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.101757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.0.96.193:32871)
0620 23:32:54.128336 (+ 26579us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.128343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.128368 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:32:54.128449 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.128454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.128648 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.128859 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:54.129777 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.131208 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.133764 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.135361 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.135523 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.135839 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.136488 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.136686 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26416,"thread_start_us":26249,"threads_started":1}
I20260620 23:32:54.137305 16971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.101275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37052)
0620 23:32:54.126372 (+ 25097us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.126436 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:54.126683 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:54.128871 (+ 2188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:54.128878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:54.128908 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:54.129532 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.129552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.131410 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.131419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.133593 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.133606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.135774 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.135784 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.136068 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.136149 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 23:32:54.136319 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 23:32:54.136689 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24761,"thread_start_us":29338,"threads_started":1}
I20260620 23:32:54.139308 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.101335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:35013 (local address 127.0.0.1:42420)
0620 23:32:54.129726 (+ 28391us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.129746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:54.129852 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:54.131055 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:54.131061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:54.131087 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:54.131564 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.131584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.133830 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.133837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.136183 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.136197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.138342 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.138349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.138565 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.138598 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:32:54.138689 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 23:32:54.138847 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28225,"thread_start_us":28076,"threads_started":1}
I20260620 23:32:54.139679 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.101965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.0.96.195:35013)
0620 23:32:54.129838 (+ 27873us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.129849 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.129898 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:32:54.130055 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.130065 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.130406 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.130659 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:54.131871 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.133675 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.136416 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.138051 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.138236 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.139011 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.139095 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.139277 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27619,"thread_start_us":27704,"threads_started":1}
I20260620 23:32:54.140539 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.102190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.0.96.194:34011)
0620 23:32:54.125291 (+ 23101us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.125306 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.130579 (+ 5273us) server_negotiation.cc:407] Connection header received
0620 23:32:54.130936 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.130945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.131184 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.131380 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:54.132636 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.134100 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.136587 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.138658 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.138824 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.139062 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.139938 (+ 876us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.140163 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22843,"thread_start_us":23126,"threads_started":1}
I20260620 23:32:54.141358 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.101609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34011 (local address 127.0.0.1:51958)
0620 23:32:54.130378 (+ 28769us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.130398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:54.130579 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:54.131641 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:54.131651 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:54.131688 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:54.132168 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.132181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.134268 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.134276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.136397 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.136410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.139262 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.139269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.139675 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.139716 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 23:32:54.140739 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0620 23:32:54.140937 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28636,"thread_start_us":25340,"threads_started":1}
I20260620 23:32:54.143409 16629 tablet_service.cc:1467] Processing CreateTablet for tablet 396c879aa9cb41a6a95d9d56c8411d54 (DEFAULT_TABLE table=test-table [id=d726379ae1734e90956885ad9efaadf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:54.144820 16629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396c879aa9cb41a6a95d9d56c8411d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:54.144788 16893 tablet_service.cc:1467] Processing CreateTablet for tablet 396c879aa9cb41a6a95d9d56c8411d54 (DEFAULT_TABLE table=test-table [id=d726379ae1734e90956885ad9efaadf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:54.145237 16761 tablet_service.cc:1467] Processing CreateTablet for tablet 396c879aa9cb41a6a95d9d56c8411d54 (DEFAULT_TABLE table=test-table [id=d726379ae1734e90956885ad9efaadf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:32:54.146653 16761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396c879aa9cb41a6a95d9d56c8411d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:54.146641 16893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 396c879aa9cb41a6a95d9d56c8411d54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:32:54.160779 16975 tablet_bootstrap.cc:492] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Bootstrap starting.
I20260620 23:32:54.163820 16975 tablet_bootstrap.cc:654] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:54.165228 16975 log.cc:824] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:54.166826 16977 tablet_bootstrap.cc:492] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Bootstrap starting.
I20260620 23:32:54.167233 16976 tablet_bootstrap.cc:492] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Bootstrap starting.
I20260620 23:32:54.170301 16975 tablet_bootstrap.cc:492] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: No bootstrap required, opened a new log
I20260620 23:32:54.170697 16975 ts_tablet_manager.cc:1397] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260620 23:32:54.171134 16977 tablet_bootstrap.cc:654] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:54.171124 16976 tablet_bootstrap.cc:654] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Neither blocks nor log segments found. Creating new log.
I20260620 23:32:54.172435 16977 log.cc:824] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:54.172435 16976 log.cc:824] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Log is configured to *not* fsync() on all Append() calls
I20260620 23:32:54.178141 16976 tablet_bootstrap.cc:492] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: No bootstrap required, opened a new log
I20260620 23:32:54.178174 16977 tablet_bootstrap.cc:492] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: No bootstrap required, opened a new log
I20260620 23:32:54.178467 16976 ts_tablet_manager.cc:1397] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260620 23:32:54.178520 16977 ts_tablet_manager.cc:1397] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20260620 23:32:54.180625 16975 raft_consensus.cc:348] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.181241 16975 raft_consensus.cc:374] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:54.181371 16975 raft_consensus.cc:729] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c986ab972558408b871105ebe5bea8bb, State: Initialized, Role: FOLLOWER
I20260620 23:32:54.182137 16975 consensus_queue.cc:260] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.184290 16975 ts_tablet_manager.cc:1428] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260620 23:32:54.188120 16976 raft_consensus.cc:348] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.188607 16976 raft_consensus.cc:374] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:54.188732 16976 raft_consensus.cc:729] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100be9b823c5407bb6999f7850efc749, State: Initialized, Role: FOLLOWER
I20260620 23:32:54.189420 16976 consensus_queue.cc:260] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.191092 16977 raft_consensus.cc:348] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.191733 16977 raft_consensus.cc:374] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:32:54.191905 16977 raft_consensus.cc:729] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d9f3d7b1f3a4781938e397efa70db26, State: Initialized, Role: FOLLOWER
I20260620 23:32:54.192785 16977 consensus_queue.cc:260] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.193396 16958 heartbeater.cc:503] Master 127.0.96.254:36397 was elected leader, sending a full tablet report...
I20260620 23:32:54.194191 16976 ts_tablet_manager.cc:1428] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260620 23:32:54.194424 16826 heartbeater.cc:503] Master 127.0.96.254:36397 was elected leader, sending a full tablet report...
I20260620 23:32:54.195153 16977 ts_tablet_manager.cc:1428] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260620 23:32:54.219862 16827 tablet.cc:2369] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:54.250082 16982 raft_consensus.cc:482] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:32:54.250550 16982 raft_consensus.cc:504] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.252454 16982 leader_election.cc:288] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c986ab972558408b871105ebe5bea8bb (127.0.96.193:32871), 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011)
I20260620 23:32:54.259586 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.252958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:55179 (local address 127.0.96.194:34011)
0620 23:32:54.253319 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.253328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.253584 (+ 256us) server_negotiation.cc:407] Connection header received
0620 23:32:54.253927 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.253933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.254036 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.254176 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:54.254796 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.255614 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.257562 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.258421 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.258677 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.258771 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.259094 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.259253 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 23:32:54.259608 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.253158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34011 (local address 127.0.96.195:55179)
0620 23:32:54.253443 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.253466 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:54.253641 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:54.254333 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:54.254345 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:54.254383 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:54.254616 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.254626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.255778 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.255786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.257359 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.257375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.258665 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.258672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.258902 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.258930 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:32:54.259062 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:32:54.259231 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:32:54.260126 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.253282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:47945 (local address 127.0.96.193:32871)
0620 23:32:54.253489 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.253496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.254687 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 23:32:54.254903 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.254907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.254968 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.255075 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:32:54.255643 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.256380 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.258283 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.259082 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.259470 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.259590 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.259723 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.259850 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:32:54.260250 16984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.253771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.96.195:47945)
0620 23:32:54.254574 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.254591 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:32:54.254743 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:32:54.255101 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:32:54.255105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:32:54.255130 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:32:54.255474 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.255483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.256542 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.256548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.258114 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:32:54.258126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.259316 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.259321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:32:54.259539 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.259575 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:32:54.259704 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 23:32:54.259851 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":418,"threads_started":1}
W20260620 23:32:54.260766 16959 tablet.cc:2369] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:32:54.261261 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396c879aa9cb41a6a95d9d56c8411d54" candidate_uuid: "100be9b823c5407bb6999f7850efc749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9f3d7b1f3a4781938e397efa70db26" is_pre_election: true
I20260620 23:32:54.261500 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396c879aa9cb41a6a95d9d56c8411d54" candidate_uuid: "100be9b823c5407bb6999f7850efc749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c986ab972558408b871105ebe5bea8bb" is_pre_election: true
I20260620 23:32:54.261987 16781 raft_consensus.cc:2454] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100be9b823c5407bb6999f7850efc749 in term 0.
I20260620 23:32:54.262138 16649 raft_consensus.cc:2454] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 100be9b823c5407bb6999f7850efc749 in term 0.
I20260620 23:32:54.262972 16847 leader_election.cc:302] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 100be9b823c5407bb6999f7850efc749, 5d9f3d7b1f3a4781938e397efa70db26, c986ab972558408b871105ebe5bea8bb; no voters:
I20260620 23:32:54.263479 16982 raft_consensus.cc:2781] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:32:54.263638 16982 raft_consensus.cc:482] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:32:54.263777 16982 raft_consensus.cc:3037] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:54.266599 16982 raft_consensus.cc:504] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.267573 16982 leader_election.cc:288] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [CANDIDATE]: Term 1 election: Requested vote from peers c986ab972558408b871105ebe5bea8bb (127.0.96.193:32871), 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011)
I20260620 23:32:54.268450 16649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396c879aa9cb41a6a95d9d56c8411d54" candidate_uuid: "100be9b823c5407bb6999f7850efc749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c986ab972558408b871105ebe5bea8bb"
I20260620 23:32:54.268460 16781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "396c879aa9cb41a6a95d9d56c8411d54" candidate_uuid: "100be9b823c5407bb6999f7850efc749" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d9f3d7b1f3a4781938e397efa70db26"
I20260620 23:32:54.268769 16781 raft_consensus.cc:3037] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:54.268741 16649 raft_consensus.cc:3037] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:32:54.271679 16781 raft_consensus.cc:2454] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100be9b823c5407bb6999f7850efc749 in term 1.
I20260620 23:32:54.272207 16649 raft_consensus.cc:2454] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 100be9b823c5407bb6999f7850efc749 in term 1.
I20260620 23:32:54.272265 16848 leader_election.cc:302] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100be9b823c5407bb6999f7850efc749, 5d9f3d7b1f3a4781938e397efa70db26; no voters:
I20260620 23:32:54.272729 16982 raft_consensus.cc:2781] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:32:54.273429 16982 raft_consensus.cc:686] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 1 LEADER]: Becoming Leader. State: Replica: 100be9b823c5407bb6999f7850efc749, State: Running, Role: LEADER
I20260620 23:32:54.274102 16982 consensus_queue.cc:237] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } }
I20260620 23:32:54.280606 16509 catalog_manager.cc:5495] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 reported cstate change: term changed from 0 to 1, leader changed from <none> to 100be9b823c5407bb6999f7850efc749 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "100be9b823c5407bb6999f7850efc749" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100be9b823c5407bb6999f7850efc749" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 35013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 } health_report { overall_health: UNKNOWN } } }
I20260620 23:32:54.320698 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.314816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.0.96.195:35013)
0620 23:32:54.315049 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.315054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.315076 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:32:54.315175 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.315179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.315293 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.315433 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:32:54.315864 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.316676 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.318313 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.318732 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.318836 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.319364 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:32:54.319919 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:32:54.320034 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.320245 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.320316 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:32:54.330732 16781 raft_consensus.cc:1261] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Refusing update from remote peer 100be9b823c5407bb6999f7850efc749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:54.330773 16649 raft_consensus.cc:1261] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Refusing update from remote peer 100be9b823c5407bb6999f7850efc749: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:32:54.332029 16985 consensus_queue.cc:1035] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c986ab972558408b871105ebe5bea8bb" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 32871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:54.332577 16982 consensus_queue.cc:1035] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d9f3d7b1f3a4781938e397efa70db26" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:32:54.369818 16509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:54.370066 16509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:32:54.374826 16509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Sending DeleteTablet for 3 replicas of tablet 396c879aa9cb41a6a95d9d56c8411d54
I20260620 23:32:54.376281 16629 tablet_service.cc:1514] Processing DeleteTablet for tablet 396c879aa9cb41a6a95d9d56c8411d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:54 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:37052
I20260620 23:32:54.376281 16761 tablet_service.cc:1514] Processing DeleteTablet for tablet 396c879aa9cb41a6a95d9d56c8411d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:54 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:51958
I20260620 23:32:54.376463 16893 tablet_service.cc:1514] Processing DeleteTablet for tablet 396c879aa9cb41a6a95d9d56c8411d54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:32:54 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42420
I20260620 23:32:54.379562 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16546
I20260620 23:32:54.387372 16999 tablet_replica.cc:330] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: stopping tablet replica
I20260620 23:32:54.388021 16999 raft_consensus.cc:2229] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:54.388748 16999 raft_consensus.cc:2258] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:54.389422 16998 tablet_replica.cc:330] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: stopping tablet replica
I20260620 23:32:54.390241 16998 raft_consensus.cc:2229] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:32:54.390906 16998 raft_consensus.cc:2258] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:54.393076 16999 ts_tablet_manager.cc:1905] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:54.393339 16998 ts_tablet_manager.cc:1905] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:54.399473 17000 tablet_replica.cc:330] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: stopping tablet replica
I20260620 23:32:54.401466 17000 raft_consensus.cc:2229] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:32:54.402412 17000 raft_consensus.cc:2258] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:32:54.402537 16999 ts_tablet_manager.cc:1918] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:54.402925 16999 log.cc:1196] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/396c879aa9cb41a6a95d9d56c8411d54
I20260620 23:32:54.403057 16972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.381008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.96.193:32871)
0620 23:32:54.381250 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.381259 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.381289 (+ 30us) server_negotiation.cc:407] Connection header received
0620 23:32:54.381374 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.381383 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.381532 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.381700 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:54.384433 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.386137 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.388265 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.388964 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.389142 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.393239 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:54.393257 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:54.393274 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:54.393373 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:54.398266 (+ 4893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.399373 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.399388 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.399393 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.399596 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.400061 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.400067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.400077 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.400624 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:54.400927 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.402185 (+ 1258us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.402464 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:32:54.403672 16999 ts_tablet_manager.cc:1939] T 396c879aa9cb41a6a95d9d56c8411d54 P c986ab972558408b871105ebe5bea8bb: Deleting consensus metadata
I20260620 23:32:54.404157 16689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:54.405612 16998 ts_tablet_manager.cc:1918] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:54.405905 16998 log.cc:1196] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/396c879aa9cb41a6a95d9d56c8411d54
I20260620 23:32:54.406824 16998 ts_tablet_manager.cc:1939] T 396c879aa9cb41a6a95d9d56c8411d54 P 5d9f3d7b1f3a4781938e397efa70db26: Deleting consensus metadata
I20260620 23:32:54.408331 17000 ts_tablet_manager.cc:1905] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:32:54.409793 16495 catalog_manager.cc:4841] TS 5d9f3d7b1f3a4781938e397efa70db26 (127.0.96.194:34011): tablet 396c879aa9cb41a6a95d9d56c8411d54 (table test-table [id=d726379ae1734e90956885ad9efaadf5]) successfully deleted
I20260620 23:32:54.417902 17000 ts_tablet_manager.cc:1918] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:32:54.418179 17000 log.cc:1196] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/396c879aa9cb41a6a95d9d56c8411d54
I20260620 23:32:54.418895 17000 ts_tablet_manager.cc:1939] T 396c879aa9cb41a6a95d9d56c8411d54 P 100be9b823c5407bb6999f7850efc749: Deleting consensus metadata
I20260620 23:32:54.421437 16496 catalog_manager.cc:4841] TS 100be9b823c5407bb6999f7850efc749 (127.0.96.195:35013): tablet 396c879aa9cb41a6a95d9d56c8411d54 (table test-table [id=d726379ae1734e90956885ad9efaadf5]) successfully deleted
I20260620 23:32:54.513932 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16546
I20260620 23:32:54.533332 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16698
W20260620 23:32:54.533510 16494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:32871 (error 108)
I20260620 23:32:54.535738 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.535289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37064)
0620 23:32:54.535500 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.535594 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260620 23:32:54.536010 16494 catalog_manager.cc:4584] TS c986ab972558408b871105ebe5bea8bb (127.0.96.193:32871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 396c879aa9cb41a6a95d9d56c8411d54: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
I20260620 23:32:54.552721 16974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.534566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.0.96.194:34011)
0620 23:32:54.534812 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.534821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.534853 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:32:54.534956 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.534964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.535090 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.535246 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:54.538129 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.539074 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.541378 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.541824 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.541970 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.546051 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:54.546069 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:54.546086 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:54.546145 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:54.549369 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.550499 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.550505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.550508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.550657 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.551149 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.551152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.551154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.551547 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:54.551769 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.552235 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.552418 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 23:32:54.553920 16821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:54.589713 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.589273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37070)
0620 23:32:54.589473 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.589569 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:32:54.590812 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.590452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37082)
0620 23:32:54.590593 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.590697 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 23:32:54.651927 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.651191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37096)
0620 23:32:54.651538 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.651697 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260620 23:32:54.653599 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.652962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37106)
0620 23:32:54.653292 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.653401 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 23:32:54.688134 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16698
I20260620 23:32:54.706514 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16830
I20260620 23:32:54.726173 16973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.707447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.96.195:35013)
0620 23:32:54.707692 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 23:32:54.707700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:32:54.707771 (+ 71us) server_negotiation.cc:407] Connection header received
0620 23:32:54.708041 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:32:54.708045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:32:54.708118 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:32:54.708244 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:32:54.710417 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.711277 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.713632 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:32:54.714134 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:32:54.714252 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:32:54.718241 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:32:54.718256 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:32:54.718272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:32:54.718315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:32:54.722334 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.723542 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.723550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.723552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.723749 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:32:54.724291 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:32:54.724295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:32:54.724297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:32:54.724700 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:32:54.725096 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0620 23:32:54.725540 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 23:32:54.725748 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 23:32:54.727191 16953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:54.735494 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.734796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37116)
0620 23:32:54.735038 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.735288 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 23:32:54.737104 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.736538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37132)
0620 23:32:54.736780 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.736863 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260620 23:32:54.855690 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16830
I20260620 23:32:54.876874 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 16476
I20260620 23:32:54.878397 16539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:32:54.994563 16969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:32:54.993916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:32871 (local address 127.0.0.1:37136)
0620 23:32:54.994174 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:32:54.994332 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:32871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 23:32:54.994746 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 16476
I20260620 23:33:15.018383 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36397
--webserver_interface=127.0.96.254
--webserver_port=43799
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:15.408502 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:15.408850 17006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:15.409030 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:15.416749 17006 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 23:33:15.416882 17006 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 23:33:15.416924 17006 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 23:33:15.417045 17006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:15.417143 17006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:15.417243 17006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:15.417313 17006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:15.417397 17006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:15.428444 17006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=43799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:15.429945 17006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:15.431416 17006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:15.446363 17012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:15.446482 17014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:15.446467 17011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:15.447605 17006 server_base.cc:1034] running on GCE node
I20260620 23:33:15.448547 17006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:15.450307 17006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:15.451634 17006 hybrid_clock.cc:648] HybridClock initialized: now 1781998395451560 us; error 59 us; skew 500 ppm
I20260620 23:33:15.452080 17006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:15 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:15.455940 17006 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:33:15.457923 17006 webserver.cc:460] Webserver started at http://127.0.96.254:43799/ using document root <none> and password file <none>
I20260620 23:33:15.458742 17006 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:15.458889 17006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:15.465831 17006 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 23:33:15.469724 17021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:15.471809 17006 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:33:15.472113 17006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "e2c547b6250e4f3ebc29b3f97a426945"
format_stamp: "Formatted at 2026-06-20 23:32:28 on dist-test-slave-v34r"
server_key: "daedfc3c751a8cce5f3f8e711ee3ab80"
server_key_iv: "932f7d56e402e4e85ca1404f596dfcaa"
server_key_version: "encryptionkey@0"
I20260620 23:33:15.473311 17006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:15.487574 17006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:15.492347 17006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:15.492668 17006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:15.514757 17006 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36397
I20260620 23:33:15.514777 17072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36397 every 8 connection(s)
I20260620 23:33:15.516283 17006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:15.517707 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17006
I20260620 23:33:15.518934 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:32871
--local_ip_for_outbound_sockets=127.0.96.193
--tserver_master_addrs=127.0.96.254:36397
--webserver_port=42123
--webserver_interface=127.0.96.193
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:33:15.528868 17073 sys_catalog.cc:263] Verifying existing consensus state
I20260620 23:33:15.534065 17073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap starting.
I20260620 23:33:15.562175 17073 log.cc:824] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:15.582854 17073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 23:33:15.583601 17073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Bootstrap complete.
I20260620 23:33:15.594673 17073 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:33:15.595820 17073 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c547b6250e4f3ebc29b3f97a426945, State: Initialized, Role: FOLLOWER
I20260620 23:33:15.596678 17073 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:33:15.596977 17073 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:15.597141 17073 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:15.597452 17073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 2 FOLLOWER]: Advancing to term 3
I20260620 23:33:15.602487 17073 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:33:15.603236 17073 leader_election.cc:302] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c547b6250e4f3ebc29b3f97a426945; no voters:
I20260620 23:33:15.604100 17073 leader_election.cc:288] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 23:33:15.604269 17077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 23:33:15.605930 17077 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [term 3 LEADER]: Becoming Leader. State: Replica: e2c547b6250e4f3ebc29b3f97a426945, State: Running, Role: LEADER
I20260620 23:33:15.606724 17077 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } }
I20260620 23:33:15.607069 17073 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:15.610543 17078 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e2c547b6250e4f3ebc29b3f97a426945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } } }
I20260620 23:33:15.613271 17078 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:15.610267 17079 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c547b6250e4f3ebc29b3f97a426945. Latest consensus state: current_term: 3 leader_uuid: "e2c547b6250e4f3ebc29b3f97a426945" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c547b6250e4f3ebc29b3f97a426945" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36397 } } }
I20260620 23:33:15.614099 17079 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:15.619405 17083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:15.631086 17083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:15.633165 17083 catalog_manager.cc:1259] Loaded cluster ID: a9e2a91f5da241ba8f2e9e9de78477cc
I20260620 23:33:15.633399 17083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:15.638480 17083 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:15.640852 17083 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Loaded TSK: 0
I20260620 23:33:15.641214 17083 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e2c547b6250e4f3ebc29b3f97a426945: Loaded TSK: 1
F20260620 23:33:15.642750 17083 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9a300819ad google::LogMessage::Fail() at ??:0
@ 0x7f9a3008488a google::LogMessage::SendToLog() at ??:0
@ 0x7f9a300814de google::LogMessage::Flush() at ??:0
@ 0x7f9a30082a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9a33de57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9a33dd473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9a45ab01c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9a45aaf783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9a45b3b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9a45b3b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9a45b855e9 std::function<>::operator()() at ??:0
@ 0x7f9a45ab3cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9a45ab1c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9a45b39610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9a45b39439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9a32ccd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a32cd091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9a32cd0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9a32cabeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a35f856db start_thread at ??:0
@ 0x7f9a2ef1b61f clone at ??:0
W20260620 23:33:15.921459 17075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:15.921806 17075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:15.922003 17075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:15.929860 17075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:15.930050 17075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:15.930258 17075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:15.940590 17075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:32871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=42123
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:15.942072 17075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:15.943490 17075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:15.959393 17121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:15.959407 17123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:15.959403 17120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:15.960016 17075 server_base.cc:1034] running on GCE node
I20260620 23:33:15.960866 17075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:15.962098 17075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:15.965549 17075 hybrid_clock.cc:648] HybridClock initialized: now 1781998395965452 us; error 95 us; skew 500 ppm
I20260620 23:33:15.966007 17075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:15 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:15.975872 17075 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:33:15.978263 17075 webserver.cc:460] Webserver started at http://127.0.96.193:42123/ using document root <none> and password file <none>
I20260620 23:33:15.979305 17075 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:15.979485 17075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:15.987421 17075 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 23:33:15.992372 17130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:15.994858 17075 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260620 23:33:15.995146 17075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "c986ab972558408b871105ebe5bea8bb"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "7e603246419aae758fa8ddcd9c874ca4"
server_key_iv: "a68d695b3c61f68fd050d887f1593994"
server_key_version: "encryptionkey@0"
I20260620 23:33:15.996497 17075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:16.039721 17075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:16.046023 17075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:16.046394 17075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:16.047730 17075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:16.049817 17075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:16.049975 17075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:16.050155 17075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:16.050266 17075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:16.092602 17075 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:32871
I20260620 23:33:16.092670 17242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:32871 every 8 connection(s)
I20260620 23:33:16.094797 17075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:16.101408 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:16.098871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:51267)
0620 23:33:16.099909 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:16.100234 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":408,"threads_started":1}
W20260620 23:33:16.102362 17243 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:36397 (0 consecutive failures): Network error: Failed to ping master at 127.0.96.254:36397: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
I20260620 23:33:16.103600 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:16.103198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:46035)
0620 23:33:16.103386 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:16.103481 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 23:33:16.104669 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17075
I20260620 23:33:16.104722 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:16.104369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:45385)
0620 23:33:16.104531 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:16.104612 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 23:33:16.105113 17243 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:33:16.105563 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:34011
--local_ip_for_outbound_sockets=127.0.96.194
--tserver_master_addrs=127.0.96.254:36397
--webserver_port=44629
--webserver_interface=127.0.96.194
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:16.455047 17246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:16.455360 17246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:16.455497 17246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:16.463227 17246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:16.463349 17246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:16.463524 17246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:16.473775 17246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:34011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=44629
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:16.475178 17246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:16.476519 17246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:16.490999 17251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:16.491117 17252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:16.492156 17254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:16.493054 17246 server_base.cc:1034] running on GCE node
I20260620 23:33:16.493753 17246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:16.494877 17246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:16.496138 17246 hybrid_clock.cc:648] HybridClock initialized: now 1781998396496093 us; error 34 us; skew 500 ppm
I20260620 23:33:16.496490 17246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:16 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:16.500290 17246 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:33:16.502362 17246 webserver.cc:460] Webserver started at http://127.0.96.194:44629/ using document root <none> and password file <none>
I20260620 23:33:16.503237 17246 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:16.503403 17246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:16.510169 17246 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:16.513937 17261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:16.515762 17246 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:33:16.516014 17246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "5d9f3d7b1f3a4781938e397efa70db26"
format_stamp: "Formatted at 2026-06-20 23:32:29 on dist-test-slave-v34r"
server_key: "cd0fded913c8adfbac585c1ddb3438f0"
server_key_iv: "27134c282b00bd8785452ebb063cac89"
server_key_version: "encryptionkey@0"
I20260620 23:33:16.517261 17246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:16.533952 17246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:16.539216 17246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:16.539598 17246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:16.541146 17246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:16.542968 17246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:16.543126 17246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:16.543285 17246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:16.543386 17246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:16.586951 17246 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34011
I20260620 23:33:16.587042 17373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34011 every 8 connection(s)
I20260620 23:33:16.588769 17246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:16.595283 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17246
I20260620 23:33:16.595877 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:16.592724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:46447)
0620 23:33:16.593895 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:16.594204 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":472,"threads_started":1}
I20260620 23:33:16.596216 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:35013
--local_ip_for_outbound_sockets=127.0.96.195
--tserver_master_addrs=127.0.96.254:36397
--webserver_port=35333
--webserver_interface=127.0.96.195
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:16.596922 17374 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:36397 (0 consecutive failures): Network error: Failed to ping master at 127.0.96.254:36397: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
I20260620 23:33:16.598321 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:16.597684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:56875)
0620 23:33:16.597970 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:16.598122 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 23:33:16.599668 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:16.599127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:39839)
0620 23:33:16.599344 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:16.599469 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 23:33:16.600023 17374 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 23:33:16.938524 17377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:16.938868 17377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:16.939013 17377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:16.946667 17377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:16.946806 17377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:16.946997 17377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:16.957300 17377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:35013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=35333
--tserver_master_addrs=127.0.96.254:36397
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:16.958709 17377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:16.960126 17377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:16.975263 17382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:16.975342 17385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:16.975658 17383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:16.977006 17377 server_base.cc:1034] running on GCE node
I20260620 23:33:16.977957 17377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:16.979311 17377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:16.980688 17377 hybrid_clock.cc:648] HybridClock initialized: now 1781998396980514 us; error 73 us; skew 500 ppm
I20260620 23:33:16.981308 17377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:16 dist-test-slave-v34r krb5kdc[15924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:16.985647 17377 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:33:16.987747 17377 webserver.cc:460] Webserver started at http://127.0.96.195:35333/ using document root <none> and password file <none>
I20260620 23:33:16.988768 17377 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:16.988961 17377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:16.996867 17377 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 23:33:17.001147 17392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:17.003391 17377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:17.003690 17377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "100be9b823c5407bb6999f7850efc749"
format_stamp: "Formatted at 2026-06-20 23:32:30 on dist-test-slave-v34r"
server_key: "32319f2060584ad883909f95bf6064dd"
server_key_iv: "99a1471214a7e4989f5a7c5a663180d5"
server_key_version: "encryptionkey@0"
I20260620 23:33:17.005138 17377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:17.027659 17377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:17.034492 17377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:17.034883 17377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:17.036269 17377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:17.038335 17377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:17.038517 17377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:17.038700 17377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:17.038818 17377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:17.080883 17377 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:35013
I20260620 23:33:17.081056 17504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:35013 every 8 connection(s)
I20260620 23:33:17.082909 17377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:17.088773 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17377
I20260620 23:33:17.089733 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:17.087086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:54289)
0620 23:33:17.088201 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:17.088437 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":423,"threads_started":1}
W20260620 23:33:17.090884 17505 heartbeater.cc:644] Failed to heartbeat to 127.0.96.254:36397 (0 consecutive failures): Network error: Failed to ping master at 127.0.96.254:36397: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
I20260620 23:33:17.092111 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:17.091606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:36495)
0620 23:33:17.091819 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:17.091927 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 23:33:17.093339 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:17.092852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:44761)
0620 23:33:17.093082 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:17.093186 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 23:33:17.093729 17505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 23:33:17.107101 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:17.106035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:44205)
0620 23:33:17.106727 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:17.106842 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":387,"threads_started":1}
I20260620 23:33:17.602182 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:17.600844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:60215)
0620 23:33:17.601595 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:17.601842 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260620 23:33:18.095523 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:18.094541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:56475)
0620 23:33:18.095156 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:18.095279 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":318,"threads_started":1}
I20260620 23:33:18.109665 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:18.108528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:60421)
0620 23:33:18.109217 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:18.109354 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":296,"threads_started":1}
I20260620 23:33:18.604590 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:18.603618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:41951)
0620 23:33:18.604260 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:18.604385 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":318,"threads_started":1}
I20260620 23:33:19.097970 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:19.096924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:56427)
0620 23:33:19.097646 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:19.097756 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":330,"threads_started":1}
I20260620 23:33:19.111873 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:19.110882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:34437)
0620 23:33:19.111549 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:19.111658 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":363,"threads_started":1}
I20260620 23:33:19.606801 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:19.605883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:47539)
0620 23:33:19.606456 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:19.606573 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":281,"threads_started":1}
I20260620 23:33:20.100023 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:20.099168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:58349)
0620 23:33:20.099702 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:20.099792 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":311,"threads_started":1}
I20260620 23:33:20.114363 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:20.113262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:56275)
0620 23:33:20.113934 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:20.114047 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":335,"threads_started":1}
I20260620 23:33:20.609383 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:20.607971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:44297)
0620 23:33:20.608802 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:20.609016 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":353,"threads_started":1}
I20260620 23:33:21.102319 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:21.101306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:33047)
0620 23:33:21.101997 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:21.102107 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":321,"threads_started":1}
I20260620 23:33:21.116580 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:21.115561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:38849)
0620 23:33:21.116221 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:21.116353 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":328,"threads_started":1}
I20260620 23:33:21.612030 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:21.610939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:52939)
0620 23:33:21.611691 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:21.611806 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":451,"threads_started":1}
I20260620 23:33:22.104645 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:22.103733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:45915)
0620 23:33:22.104339 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:22.104447 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":288,"threads_started":1}
I20260620 23:33:22.118876 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:22.117914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:59795)
0620 23:33:22.118515 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:22.118639 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":317,"threads_started":1}
I20260620 23:33:22.614464 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:22.613342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:50041)
0620 23:33:22.614029 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:22.614200 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":335,"threads_started":1}
I20260620 23:33:23.107071 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:23.106031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:40991)
0620 23:33:23.106793 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:23.106895 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":509,"threads_started":1}
I20260620 23:33:23.121248 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:23.120086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:49837)
0620 23:33:23.120727 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:23.120847 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":279,"threads_started":1}
I20260620 23:33:23.616595 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:23.615603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:53685)
0620 23:33:23.616277 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:23.616373 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":340,"threads_started":1}
I20260620 23:33:24.109797 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:24.108559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:53195)
0620 23:33:24.109363 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:24.109477 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":429,"threads_started":1}
I20260620 23:33:24.123381 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:24.122519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:39807)
0620 23:33:24.123076 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:24.123180 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":283,"threads_started":1}
I20260620 23:33:24.618755 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:24.617950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:41183)
0620 23:33:24.618492 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:24.618588 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
I20260620 23:33:25.111928 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:25.111066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:46915)
0620 23:33:25.111633 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:25.111740 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":293,"threads_started":1}
I20260620 23:33:25.126025 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:25.124699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:48281)
0620 23:33:25.125537 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:25.125719 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":445,"threads_started":1}
I20260620 23:33:25.620873 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:25.619844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:51789)
0620 23:33:25.620581 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:25.620677 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":355,"threads_started":1}
I20260620 23:33:26.114045 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:26.113125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:46375)
0620 23:33:26.113699 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:26.113823 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":269,"threads_started":1}
I20260620 23:33:26.128237 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:26.127307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:43111)
0620 23:33:26.127926 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:26.128043 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":329,"threads_started":1}
I20260620 23:33:26.623526 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:26.622239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:33263)
0620 23:33:26.623161 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:26.623286 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":416,"threads_started":1}
I20260620 23:33:27.116171 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:27.115396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:33639)
0620 23:33:27.115887 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:27.115985 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
I20260620 23:33:27.130609 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:27.129614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:39483)
0620 23:33:27.130130 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:27.130234 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":260,"threads_started":1}
I20260620 23:33:27.625797 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:27.624836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:49575)
0620 23:33:27.625491 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:27.625591 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":375,"threads_started":1}
I20260620 23:33:28.118570 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:28.117494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:50909)
0620 23:33:28.118111 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:28.118332 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":305,"threads_started":1}
I20260620 23:33:28.132534 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:28.131828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:35357)
0620 23:33:28.132284 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:28.132371 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260620 23:33:28.627965 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:28.627120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:44425)
0620 23:33:28.627682 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:28.627792 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":296,"threads_started":1}
I20260620 23:33:29.120760 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:29.119902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:45895)
0620 23:33:29.120470 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:29.120580 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":266,"threads_started":1}
I20260620 23:33:29.135012 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:29.133906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:36771)
0620 23:33:29.134654 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:29.134779 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":368,"threads_started":1}
I20260620 23:33:29.630146 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:29.629200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:49837)
0620 23:33:29.629821 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:29.629940 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":327,"threads_started":1}
I20260620 23:33:30.123042 17547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:30.122042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:43023)
0620 23:33:30.122724 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:30.122846 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":286,"threads_started":1}
I20260620 23:33:30.137065 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:30.136118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:45335)
0620 23:33:30.136664 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:30.136769 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":263,"threads_started":1}
I20260620 23:33:30.632227 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:30.631147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:60511)
0620 23:33:30.631827 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:30.631953 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":295,"threads_started":1}
I20260620 23:33:31.125109 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:31.124202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:49041)
0620 23:33:31.124744 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:31.124841 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260620 23:33:31.139770 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:31.138671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.193:57591)
0620 23:33:31.139380 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:31.139522 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":363,"threads_started":1}
I20260620 23:33:31.634662 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:31.633734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.194:48629)
0620 23:33:31.634328 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:31.634458 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":307,"threads_started":1}
I20260620 23:33:32.099283 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17075
I20260620 23:33:32.120316 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:32.100858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.0.96.193:32871)
0620 23:33:32.101968 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:32.101991 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:32.102054 (+ 63us) server_negotiation.cc:407] Connection header received
0620 23:33:32.103862 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:32.103871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:32.104299 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:32.104646 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:32.106471 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:32.107718 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:32.109336 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:32.110053 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:32.110195 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:32.113619 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:32.113626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:32.113643 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:32.113693 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:32.116738 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:32.117773 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:32.117779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:32.117786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:32.117962 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:32.118383 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:32.118386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:32.118388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:32.118912 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:32.119151 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:32.119685 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 23:33:32.120020 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":619,"threads_started":1}
I20260620 23:33:32.121896 17238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:32.128077 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:32.126705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36397 (local address 127.0.96.195:34825)
0620 23:33:32.127565 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:32.127763 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.254:36397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":495,"threads_started":1}
I20260620 23:33:32.210419 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17075
I20260620 23:33:32.228515 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17246
I20260620 23:33:32.248615 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:32.229515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.0.96.194:34011)
0620 23:33:32.230138 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:32.230152 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:32.230191 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:33:32.231762 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:32.231770 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:32.232188 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:32.232488 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:32.234399 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:32.235647 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:32.237322 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:32.238009 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:32.238187 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:32.241540 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:32.241549 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:32.241566 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:32.241611 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:32.244850 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:32.245948 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:32.245955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:32.245962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:32.246180 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:32.246652 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:32.246655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:32.246657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:32.247206 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:32.247438 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:32.247952 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 23:33:32.248326 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":318,"threads_started":1}
I20260620 23:33:32.250375 17369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:32.339571 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17246
I20260620 23:33:32.357971 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17377
I20260620 23:33:32.380635 17559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:32.359294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.96.195:35013)
0620 23:33:32.360020 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:32.360039 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:32.360096 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:33:32.362178 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:32.362193 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:32.362842 (+ 649us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:32.363264 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:32.364729 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:32.366606 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:32.368080 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:32.368967 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:32.369261 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:32.372081 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:32.372093 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:32.372118 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:32.372192 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:32.376255 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:32.377201 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:32.377211 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:32.377221 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:32.377513 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:32.377976 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:32.377980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:32.377982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:32.378775 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:32.379082 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:32.379610 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 23:33:32.380235 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":327,"threads_started":1}
I20260620 23:33:32.382725 17500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:32.469550 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17377
2026-06-20T23:33:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65823 ms)
[----------] 30 tests from SecurityITest (324674 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:33:32 dist-test-slave-v34r krb5kdc[17563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:33:32 dist-test-slave-v34r krb5kdc[17563](info): set up 2 sockets
Jun 20 23:33:32 dist-test-slave-v34r krb5kdc[17563](info): commencing operation
krb5kdc: starting...
W20260620 23:33:33.597285 387 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:33:33 dist-test-slave-v34r krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:33:33.890671 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:32789
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:32789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:34.229813 17576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:34.230122 17576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:34.230212 17576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:34.237527 17576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:34.237651 17576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:34.237704 17576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:34.237756 17576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:34.237795 17576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:34.248263 17576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:32789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:32789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:34.249701 17576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:34.250964 17576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:34.263935 17584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:34.264040 17581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:34.264071 17582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:34.265297 17576 server_base.cc:1034] running on GCE node
I20260620 23:33:34.266377 17576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:34.268299 17576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:34.269632 17576 hybrid_clock.cc:648] HybridClock initialized: now 1781998414269550 us; error 50 us; skew 500 ppm
I20260620 23:33:34.270125 17576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:34 dist-test-slave-v34r krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:34.274041 17576 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:33:34.276114 17576 webserver.cc:460] Webserver started at http://127.0.96.254:46183/ using document root <none> and password file <none>
I20260620 23:33:34.277143 17576 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:34.277318 17576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:34.277698 17576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:34.281832 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "e9455d3fd8de41a9af559fcc887c3670"
format_stamp: "Formatted at 2026-06-20 23:33:34 on dist-test-slave-v34r"
server_key: "debd2aecba62c8030c1a9ad9c4221e11"
server_key_iv: "2cb1a0673cf0879eee877fcbda84ed25"
server_key_version: "encryptionkey@0"
I20260620 23:33:34.282814 17576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "e9455d3fd8de41a9af559fcc887c3670"
format_stamp: "Formatted at 2026-06-20 23:33:34 on dist-test-slave-v34r"
server_key: "debd2aecba62c8030c1a9ad9c4221e11"
server_key_iv: "2cb1a0673cf0879eee877fcbda84ed25"
server_key_version: "encryptionkey@0"
I20260620 23:33:34.289793 17576 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 23:33:34.294613 17591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:34.296509 17576 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:33:34.296789 17576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "e9455d3fd8de41a9af559fcc887c3670"
format_stamp: "Formatted at 2026-06-20 23:33:34 on dist-test-slave-v34r"
server_key: "debd2aecba62c8030c1a9ad9c4221e11"
server_key_iv: "2cb1a0673cf0879eee877fcbda84ed25"
server_key_version: "encryptionkey@0"
I20260620 23:33:34.297319 17576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:34.316489 17576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:34.321997 17576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:34.322386 17576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:34.347314 17576 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:32789
I20260620 23:33:34.347333 17642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:32789 every 8 connection(s)
I20260620 23:33:34.349231 17576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:34.351843 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17576
I20260620 23:33:34.352078 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:34.353037 387 external_mini_cluster.cc:1428] Setting key f49700c69048e2292630b0f3ee08343b
I20260620 23:33:34.354539 17643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:34.371359 17643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670: Bootstrap starting.
Jun 20 23:33:34 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:34.375495 17643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:34.377113 17643 log.cc:824] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:34.381496 17643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670: No bootstrap required, opened a new log
I20260620 23:33:34.384775 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:34.355712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.96.254:32789)
0620 23:33:34.356778 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:34.356802 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:34.356858 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:33:34.359678 (+ 2820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:34.359724 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:34.360360 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:34.360863 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:34.363020 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:34.364593 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:34.366962 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:34.367948 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:34.368182 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:34.374161 (+ 5979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:34.374177 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:34.374201 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:34.374269 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:34.379051 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:34.380184 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:34.380195 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:34.380203 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:34.380447 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:34.380946 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:34.380951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:34.380955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:34.381848 (+ 893us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:34.382211 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:34.382711 (+ 500us) server_negotiation.cc:299] Negotiation successful
0620 23:33:34.383217 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":470,"threads_started":1}
I20260620 23:33:34.391506 17643 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455d3fd8de41a9af559fcc887c3670" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 32789 } }
I20260620 23:33:34.391942 17643 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:34.392023 17643 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9455d3fd8de41a9af559fcc887c3670, State: Initialized, Role: FOLLOWER
I20260620 23:33:34.392592 17643 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455d3fd8de41a9af559fcc887c3670" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 32789 } }
I20260620 23:33:34.392812 17643 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:34.392935 17643 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:34.393153 17643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:34.395671 17643 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455d3fd8de41a9af559fcc887c3670" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 32789 } }
I20260620 23:33:34.396216 17643 leader_election.cc:302] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9455d3fd8de41a9af559fcc887c3670; no voters:
I20260620 23:33:34.397125 17643 leader_election.cc:288] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:34.397342 17648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:34.398075 17648 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [term 1 LEADER]: Becoming Leader. State: Replica: e9455d3fd8de41a9af559fcc887c3670, State: Running, Role: LEADER
I20260620 23:33:34.399029 17648 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455d3fd8de41a9af559fcc887c3670" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 32789 } }
I20260620 23:33:34.399622 17643 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:34.403765 17650 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9455d3fd8de41a9af559fcc887c3670. Latest consensus state: current_term: 1 leader_uuid: "e9455d3fd8de41a9af559fcc887c3670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455d3fd8de41a9af559fcc887c3670" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 32789 } } }
I20260620 23:33:34.404369 17650 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:34.405164 17649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9455d3fd8de41a9af559fcc887c3670" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9455d3fd8de41a9af559fcc887c3670" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 32789 } } }
I20260620 23:33:34.405550 17649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:34.408236 17656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:34.415104 17656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:34.427918 17656 catalog_manager.cc:1347] Generated new cluster ID: ce3f979508d7404f99faad50a5cc5839
I20260620 23:33:34.428172 17656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:34.441323 17656 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:34.442813 17656 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:34.451745 17656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670: Generated new TSK 0
I20260620 23:33:34.453334 17656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:33:34.524545 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:32789
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:34.906536 17669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:34.906876 17669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:34.907068 17669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:34.914475 17669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:34.914625 17669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:34.914813 17669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:34.924518 17669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:34.925953 17669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:34.927228 17669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:34.941547 17674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:34.941694 17677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:34.941843 17675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:34.948532 17669 server_base.cc:1034] running on GCE node
I20260620 23:33:34.949339 17669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:34.950457 17669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:34.951756 17669 hybrid_clock.cc:648] HybridClock initialized: now 1781998414951682 us; error 85 us; skew 500 ppm
I20260620 23:33:34.952123 17669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:34 dist-test-slave-v34r krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:34.955729 17669 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:33:34.957597 17669 webserver.cc:460] Webserver started at http://127.0.96.193:35985/ using document root <none> and password file <none>
I20260620 23:33:34.958375 17669 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:34.958513 17669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:34.958855 17669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:34.961910 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "b18e97f766644c6a9c32307bdae3ff1e"
format_stamp: "Formatted at 2026-06-20 23:33:34 on dist-test-slave-v34r"
server_key: "6f7963533c0c966eb0840bb4f9078306"
server_key_iv: "ddb905f42b14cfa68a6f46aea19b4a31"
server_key_version: "encryptionkey@0"
I20260620 23:33:34.962738 17669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "b18e97f766644c6a9c32307bdae3ff1e"
format_stamp: "Formatted at 2026-06-20 23:33:34 on dist-test-slave-v34r"
server_key: "6f7963533c0c966eb0840bb4f9078306"
server_key_iv: "ddb905f42b14cfa68a6f46aea19b4a31"
server_key_version: "encryptionkey@0"
I20260620 23:33:34.968715 17669 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260620 23:33:34.972936 17684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:34.974738 17669 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:34.975006 17669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "b18e97f766644c6a9c32307bdae3ff1e"
format_stamp: "Formatted at 2026-06-20 23:33:34 on dist-test-slave-v34r"
server_key: "6f7963533c0c966eb0840bb4f9078306"
server_key_iv: "ddb905f42b14cfa68a6f46aea19b4a31"
server_key_version: "encryptionkey@0"
I20260620 23:33:34.975255 17669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:34.998428 17669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:35.003705 17669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:35.004040 17669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:35.005285 17669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:35.007020 17669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:35.007211 17669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:35.007382 17669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:35.007463 17669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:35.046732 17669 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:44151
I20260620 23:33:35.046831 17796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:44151 every 8 connection(s)
I20260620 23:33:35.048120 17669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:35.050304 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17669
I20260620 23:33:35.050541 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:33:35.051200 387 external_mini_cluster.cc:1428] Setting key 455349791626bc449aae219ed32da92c
Jun 20 23:33:35 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:35.073503 17799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:35.051474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:44259 (local address 127.0.96.254:32789)
0620 23:33:35.052275 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:35.052283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:35.053736 (+ 1453us) server_negotiation.cc:407] Connection header received
0620 23:33:35.055980 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:35.055988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:35.056105 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:35.056234 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:35.058233 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.059478 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.061484 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.061957 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.062124 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:35.066383 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:35.066389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:35.066392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:35.066433 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:35.069650 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:35.070645 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:35.070651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:35.070654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:35.070810 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:35.071342 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:35.071354 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:35.071357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:35.071675 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:35.071854 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:35.072737 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 23:33:35.073068 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:33:35.074877 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:35.052444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:32789 (local address 127.0.96.193:44259)
0620 23:33:35.053495 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:35.053560 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:35.055629 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:35.056515 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:35.056525 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:35.057294 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:35.058024 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:35.058043 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.059679 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.059684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:35.061278 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:35.061294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.062260 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.062271 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:35.062547 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:35.063389 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:35.063416 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:35.066055 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:35.069860 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:35.069879 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:35.069900 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:35.070474 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:35.070955 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:35.070960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:35.070965 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:35.071210 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:35.071818 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:35.071828 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:35.072284 (+ 456us) client_negotiation.cc:769] Sending connection context
0620 23:33:35.072615 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 23:33:35.073339 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":452,"threads_started":1}
I20260620 23:33:35.077229 17797 heartbeater.cc:347] Connected to a master server at 127.0.96.254:32789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:33:35.077765 17797 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:35.078765 17797 heartbeater.cc:511] Master 127.0.96.254:32789 requested a full tablet report, sending...
I20260620 23:33:35.081851 17608 ts_manager.cc:194] Registered new tserver with Master: b18e97f766644c6a9c32307bdae3ff1e (127.0.96.193:44151)
I20260620 23:33:35.083873 17608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:44259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:33:35.105876 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:32789
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:35.448869 17803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:35.449268 17803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:35.449402 17803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:35.457494 17803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:35.457619 17803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:35.457738 17803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:35.467801 17803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:35.469147 17803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:35.470479 17803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:35.484838 17808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:35.484964 17809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:35.485345 17811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:35.486447 17803 server_base.cc:1034] running on GCE node
I20260620 23:33:35.487154 17803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:35.488168 17803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:35.489435 17803 hybrid_clock.cc:648] HybridClock initialized: now 1781998415489388 us; error 48 us; skew 500 ppm
I20260620 23:33:35.489827 17803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:35 dist-test-slave-v34r krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:35.493568 17803 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:33:35.495572 17803 webserver.cc:460] Webserver started at http://127.0.96.194:41765/ using document root <none> and password file <none>
I20260620 23:33:35.496354 17803 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:35.496502 17803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:35.496829 17803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:35.500021 17803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "960013f4ad2f42ef8081a99747a60b93"
format_stamp: "Formatted at 2026-06-20 23:33:35 on dist-test-slave-v34r"
server_key: "dd818be80bcf6e522371ebd732f39003"
server_key_iv: "f94d571fc0ecb06ca40add26420b1a34"
server_key_version: "encryptionkey@0"
I20260620 23:33:35.500921 17803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "960013f4ad2f42ef8081a99747a60b93"
format_stamp: "Formatted at 2026-06-20 23:33:35 on dist-test-slave-v34r"
server_key: "dd818be80bcf6e522371ebd732f39003"
server_key_iv: "f94d571fc0ecb06ca40add26420b1a34"
server_key_version: "encryptionkey@0"
I20260620 23:33:35.506973 17803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 23:33:35.511158 17818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:35.512854 17803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 23:33:35.513082 17803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "960013f4ad2f42ef8081a99747a60b93"
format_stamp: "Formatted at 2026-06-20 23:33:35 on dist-test-slave-v34r"
server_key: "dd818be80bcf6e522371ebd732f39003"
server_key_iv: "f94d571fc0ecb06ca40add26420b1a34"
server_key_version: "encryptionkey@0"
I20260620 23:33:35.513298 17803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:35.527930 17803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:35.532397 17803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:35.532727 17803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:35.533854 17803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:35.535272 17803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:35.535384 17803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:35.535527 17803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:35.535624 17803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:35.575464 17803 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:38347
I20260620 23:33:35.575659 17930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:38347 every 8 connection(s)
I20260620 23:33:35.576820 17803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:35.582834 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17803
I20260620 23:33:35.583086 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:33:35.583636 387 external_mini_cluster.cc:1428] Setting key f7aba1c221e54478095bc1fd18d9ba29
Jun 20 23:33:35 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:35.602797 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:35.580419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:55503 (local address 127.0.96.254:32789)
0620 23:33:35.581009 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:35.581017 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:35.582268 (+ 1251us) server_negotiation.cc:407] Connection header received
0620 23:33:35.584800 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:35.584807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:35.584927 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:35.585112 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:35.587306 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.588131 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.590046 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.590532 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.590694 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:35.595473 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:35.595480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:35.595483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:35.595530 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:35.598701 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:35.599739 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:35.599744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:35.599747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:35.599979 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:35.600499 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:35.600504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:35.600508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:35.600857 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:35.601072 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:35.602070 (+ 998us) server_negotiation.cc:299] Negotiation successful
0620 23:33:35.602363 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260620 23:33:35.604135 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:35.580958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:32789 (local address 127.0.96.194:55503)
0620 23:33:35.582031 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:35.582095 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:35.584391 (+ 2296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:35.585425 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:35.585435 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:35.586327 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:35.587079 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:35.587101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.588327 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.588332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:35.589863 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:35.589879 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:35.590819 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:35.590826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:35.591118 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:35.592006 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:35.592036 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:35.595105 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:35.598961 (+ 3856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:35.598972 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:35.598992 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:35.599568 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:35.600107 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:35.600114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:35.600122 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:35.600364 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:35.601083 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:35.601095 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:35.601569 (+ 474us) client_negotiation.cc:769] Sending connection context
0620 23:33:35.601955 (+ 386us) client_negotiation.cc:240] Negotiation successful
0620 23:33:35.602672 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":454,"threads_started":1}
I20260620 23:33:35.606093 17931 heartbeater.cc:347] Connected to a master server at 127.0.96.254:32789
I20260620 23:33:35.606570 17931 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:35.607782 17931 heartbeater.cc:511] Master 127.0.96.254:32789 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
I20260620 23:33:35.609825 17608 ts_manager.cc:194] Registered new tserver with Master: 960013f4ad2f42ef8081a99747a60b93 (127.0.96.194:38347)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
I20260620 23:33:35.610831 17608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:55503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:33:35.646751 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:32789
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:35.977278 17937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:35.977623 17937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:35.977797 17937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:35.985587 17937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:35.985749 17937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:35.985945 17937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:35.996205 17937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:32789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:35.997540 17937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:35.998845 17937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:36.014137 17945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:36.014014 17942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:36.014004 17943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:36.016108 17937 server_base.cc:1034] running on GCE node
I20260620 23:33:36.016947 17937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:36.018294 17937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:36.019590 17937 hybrid_clock.cc:648] HybridClock initialized: now 1781998416019486 us; error 92 us; skew 500 ppm
I20260620 23:33:36.020217 17937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:36 dist-test-slave-v34r krb5kdc[17563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:36.029567 17937 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:33:36.033222 17937 webserver.cc:460] Webserver started at http://127.0.96.195:45375/ using document root <none> and password file <none>
I20260620 23:33:36.034396 17937 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:36.034590 17937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:36.035024 17937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:36.040190 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "7252a71932284b52b2852f5a61cc1909"
format_stamp: "Formatted at 2026-06-20 23:33:36 on dist-test-slave-v34r"
server_key: "9ae7fcd813ee6c2a7647cd703955d59c"
server_key_iv: "79b3b6fd33160ba1852a8772caf7e95d"
server_key_version: "encryptionkey@0"
I20260620 23:33:36.041488 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "7252a71932284b52b2852f5a61cc1909"
format_stamp: "Formatted at 2026-06-20 23:33:36 on dist-test-slave-v34r"
server_key: "9ae7fcd813ee6c2a7647cd703955d59c"
server_key_iv: "79b3b6fd33160ba1852a8772caf7e95d"
server_key_version: "encryptionkey@0"
I20260620 23:33:36.050302 17937 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260620 23:33:36.057138 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:36.059093 17937 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260620 23:33:36.059351 17937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "7252a71932284b52b2852f5a61cc1909"
format_stamp: "Formatted at 2026-06-20 23:33:36 on dist-test-slave-v34r"
server_key: "9ae7fcd813ee6c2a7647cd703955d59c"
server_key_iv: "79b3b6fd33160ba1852a8772caf7e95d"
server_key_version: "encryptionkey@0"
I20260620 23:33:36.059617 17937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:36.088033 17797 heartbeater.cc:503] Master 127.0.96.254:32789 was elected leader, sending a full tablet report...
I20260620 23:33:36.091792 17937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:36.097910 17937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:36.098301 17937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:36.100096 17937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:36.102345 17937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:36.102488 17937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:36.102640 17937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:36.102710 17937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:36.146260 17937 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:40777
I20260620 23:33:36.146313 18064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:40777 every 8 connection(s)
I20260620 23:33:36.147895 17937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:36.157294 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 17937
I20260620 23:33:36.157487 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:33:36.158079 387 external_mini_cluster.cc:1428] Setting key b0cdd6f239c446005c6de75a137fffb6
Jun 20 23:33:36 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:36.172318 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.150982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:41265 (local address 127.0.96.254:32789)
0620 23:33:36.151486 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.151491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.152627 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 23:33:36.154662 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.154667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.154763 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.154884 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:36.156488 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.157621 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.159942 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.160539 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.160718 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.165244 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:36.165272 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:36.165279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:36.165333 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:36.168454 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.169506 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.169513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.169517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.169669 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.170170 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.170175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.170183 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.170499 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:36.170681 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.171551 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.171861 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":281,"threads_started":1}
I20260620 23:33:36.173707 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.151582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:32789 (local address 127.0.96.195:41265)
0620 23:33:36.152425 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:36.152477 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:36.154366 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:36.155138 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:36.155146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:36.155725 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:36.156316 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.156335 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.157840 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.157845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.159608 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.159624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.160824 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.160832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.161136 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.161944 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:36.161991 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:36.164879 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:36.168658 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:36.168669 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:36.168695 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:36.169362 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:36.169796 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:36.169802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:36.169806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:36.170038 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:36.170639 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:36.170648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:36.171105 (+ 457us) client_negotiation.cc:769] Sending connection context
0620 23:33:36.171433 (+ 328us) client_negotiation.cc:240] Negotiation successful
0620 23:33:36.172139 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":332,"threads_started":1}
I20260620 23:33:36.175542 18065 heartbeater.cc:347] Connected to a master server at 127.0.96.254:32789
I20260620 23:33:36.175993 18065 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:36.176870 18065 heartbeater.cc:511] Master 127.0.96.254:32789 requested a full tablet report, sending...
I20260620 23:33:36.178684 17607 ts_manager.cc:194] Registered new tserver with Master: 7252a71932284b52b2852f5a61cc1909 (127.0.96.195:40777)
I20260620 23:33:36.179858 17607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:41265
I20260620 23:33:36.187862 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:33:36.208496 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.193618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.0.96.254:32789)
0620 23:33:36.193900 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.193910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.194258 (+ 348us) server_negotiation.cc:407] Connection header received
0620 23:33:36.194464 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.194470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.194547 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.194662 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:36.196104 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.196846 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.198359 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.198782 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.198916 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.202289 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:36.202300 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:36.202303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:36.202335 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:36.205360 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.206057 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.206061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.206064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.206180 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.206550 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.206557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.206562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.207056 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:36.207311 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.207858 (+ 547us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.208137 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:33:36.214025 17607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:33:36.222299 17607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:33:36.264840 18078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.239152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:40777 (local address 127.0.0.1:53836)
0620 23:33:36.254746 (+ 15594us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:36.254780 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:36.254950 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:36.256010 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:36.256020 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:36.256081 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:36.256584 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.256613 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.258865 (+ 2252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.258872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.261179 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.261199 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.263358 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.263365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.263764 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.263842 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 23:33:36.264016 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 23:33:36.264337 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15450,"mutex_wait_us":89,"thread_start_us":15265,"threads_started":1}
I20260620 23:33:36.264925 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.239589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.0.96.195:40777)
0620 23:33:36.254375 (+ 14786us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.254389 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.255091 (+ 702us) server_negotiation.cc:407] Connection header received
0620 23:33:36.255220 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.255229 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.255589 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.255921 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:36.256937 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.258637 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.261450 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.263102 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.263299 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.263614 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.264142 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.264406 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14540,"thread_start_us":14320,"threads_started":1}
I20260620 23:33:36.266309 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.239136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42284 (local address 127.0.96.193:44151)
0620 23:33:36.255948 (+ 16812us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.255959 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.255991 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:33:36.256128 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.256139 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.256418 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.256648 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:36.257844 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.259578 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.262955 (+ 3377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.264668 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.264813 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.265120 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.265797 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.265973 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16595,"thread_start_us":16352,"threads_started":1}
I20260620 23:33:36.267398 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.238699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:44151 (local address 127.0.0.1:42284)
0620 23:33:36.252858 (+ 14159us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:36.252961 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:36.253350 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:36.256889 (+ 3539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:36.256894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:36.256925 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:36.257589 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.257606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.259751 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.259757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.262699 (+ 2942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.262716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.265287 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.265295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.265496 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.265596 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 23:33:36.266760 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0620 23:33:36.266961 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13644,"thread_start_us":15732,"threads_started":1}
I20260620 23:33:36.267774 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.256741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:38347 (local address 127.0.0.1:45896)
0620 23:33:36.257571 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:36.257595 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:36.257744 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:36.258602 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:36.258607 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:36.258641 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:36.259145 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.259155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.261210 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.261215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.263550 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.263567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.266686 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.266698 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.266896 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.266930 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:33:36.267027 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 23:33:36.267179 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":37,"thread_start_us":433,"threads_started":1}
I20260620 23:33:36.268354 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.256007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.96.194:38347)
0620 23:33:36.256736 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.256748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.257888 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 23:33:36.258020 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.258031 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.258332 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.258555 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:36.259384 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.261007 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.263768 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.265817 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.265982 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.266202 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.267763 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.268030 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":377,"threads_started":1}
I20260620 23:33:36.269749 18000 tablet_service.cc:1467] Processing CreateTablet for tablet d2ead886dea240589898b34f6705620f (DEFAULT_TABLE table=test-table [id=5a296d82d9e54de39a9cca5f87641176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:36.271390 18000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ead886dea240589898b34f6705620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:36.271050 17732 tablet_service.cc:1467] Processing CreateTablet for tablet d2ead886dea240589898b34f6705620f (DEFAULT_TABLE table=test-table [id=5a296d82d9e54de39a9cca5f87641176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:36.272405 17732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ead886dea240589898b34f6705620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:36.273334 17866 tablet_service.cc:1467] Processing CreateTablet for tablet d2ead886dea240589898b34f6705620f (DEFAULT_TABLE table=test-table [id=5a296d82d9e54de39a9cca5f87641176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:36.274914 17866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ead886dea240589898b34f6705620f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:36.290110 18084 tablet_bootstrap.cc:492] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Bootstrap starting.
I20260620 23:33:36.290905 18085 tablet_bootstrap.cc:492] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Bootstrap starting.
I20260620 23:33:36.292039 18086 tablet_bootstrap.cc:492] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Bootstrap starting.
I20260620 23:33:36.295220 18084 tablet_bootstrap.cc:654] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:36.295301 18085 tablet_bootstrap.cc:654] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:36.295826 18086 tablet_bootstrap.cc:654] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:36.296633 18084 log.cc:824] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:36.297581 18086 log.cc:824] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:36.297596 18085 log.cc:824] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:36.302520 18086 tablet_bootstrap.cc:492] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: No bootstrap required, opened a new log
I20260620 23:33:36.302850 18086 ts_tablet_manager.cc:1397] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:33:36.303614 18084 tablet_bootstrap.cc:492] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: No bootstrap required, opened a new log
I20260620 23:33:36.303612 18085 tablet_bootstrap.cc:492] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: No bootstrap required, opened a new log
I20260620 23:33:36.303972 18085 ts_tablet_manager.cc:1397] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 23:33:36.303972 18084 ts_tablet_manager.cc:1397] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260620 23:33:36.310968 18086 raft_consensus.cc:348] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.311419 18086 raft_consensus.cc:374] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:36.311507 18086 raft_consensus.cc:729] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 960013f4ad2f42ef8081a99747a60b93, State: Initialized, Role: FOLLOWER
I20260620 23:33:36.312163 18086 consensus_queue.cc:260] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.313367 17931 heartbeater.cc:503] Master 127.0.96.254:32789 was elected leader, sending a full tablet report...
I20260620 23:33:36.314067 18086 ts_tablet_manager.cc:1428] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260620 23:33:36.316308 18085 raft_consensus.cc:348] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.316308 18084 raft_consensus.cc:348] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.316892 18084 raft_consensus.cc:374] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:36.317008 18085 raft_consensus.cc:374] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:36.317052 18084 raft_consensus.cc:729] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7252a71932284b52b2852f5a61cc1909, State: Initialized, Role: FOLLOWER
I20260620 23:33:36.317108 18085 raft_consensus.cc:729] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18e97f766644c6a9c32307bdae3ff1e, State: Initialized, Role: FOLLOWER
I20260620 23:33:36.317852 18084 consensus_queue.cc:260] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.317857 18085 consensus_queue.cc:260] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.319240 18065 heartbeater.cc:503] Master 127.0.96.254:32789 was elected leader, sending a full tablet report...
I20260620 23:33:36.320474 18084 ts_tablet_manager.cc:1428] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20260620 23:33:36.322389 18085 ts_tablet_manager.cc:1428] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s
W20260620 23:33:36.329169 17932 tablet.cc:2369] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:36.329284 18091 raft_consensus.cc:482] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:33:36.329737 18091 raft_consensus.cc:504] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.331764 18091 leader_election.cc:288] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b18e97f766644c6a9c32307bdae3ff1e (127.0.96.193:44151), 960013f4ad2f42ef8081a99747a60b93 (127.0.96.194:38347)
I20260620 23:33:36.339373 18068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.332171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:44151 (local address 127.0.96.195:47617)
0620 23:33:36.332395 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:36.332415 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:36.332552 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:36.333258 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:36.333263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:36.333292 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:36.333538 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.333550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.335025 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.335031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.336709 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.336724 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.338537 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.338543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.338760 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.338800 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 23:33:36.338894 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 23:33:36.339057 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:33:36.340421 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.332360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:47617 (local address 127.0.96.193:44151)
0620 23:33:36.332595 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.332602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.332629 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:33:36.332725 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.332730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.332812 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.332990 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:36.333738 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.334850 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.336912 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.338177 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.338334 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.338415 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.340049 (+ 1634us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.340186 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:33:36.340462 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.332803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:49023 (local address 127.0.96.194:38347)
0620 23:33:36.333103 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.333110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.333505 (+ 395us) server_negotiation.cc:407] Connection header received
0620 23:33:36.333633 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.333641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.333737 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.333815 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:36.334532 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.335564 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.337745 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.339020 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.339152 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.339231 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.340073 (+ 842us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.340213 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:33:36.341202 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.332642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:38347 (local address 127.0.96.195:49023)
0620 23:33:36.333389 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:36.333402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:36.333515 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:36.333866 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:36.333871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:36.333898 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:36.334342 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.334356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.335750 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.335754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.337511 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:36.337526 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.339417 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.339423 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:36.339883 (+ 460us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.339913 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 23:33:36.340642 (+ 729us) client_negotiation.cc:240] Negotiation successful
0620 23:33:36.340796 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":424,"threads_started":1}
I20260620 23:33:36.341737 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ead886dea240589898b34f6705620f" candidate_uuid: "7252a71932284b52b2852f5a61cc1909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18e97f766644c6a9c32307bdae3ff1e" is_pre_election: true
I20260620 23:33:36.342480 17752 raft_consensus.cc:2454] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7252a71932284b52b2852f5a61cc1909 in term 0.
I20260620 23:33:36.342525 17886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ead886dea240589898b34f6705620f" candidate_uuid: "7252a71932284b52b2852f5a61cc1909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960013f4ad2f42ef8081a99747a60b93" is_pre_election: true
I20260620 23:33:36.343266 17886 raft_consensus.cc:2454] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7252a71932284b52b2852f5a61cc1909 in term 0.
I20260620 23:33:36.343292 17953 leader_election.cc:302] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7252a71932284b52b2852f5a61cc1909, b18e97f766644c6a9c32307bdae3ff1e; no voters:
I20260620 23:33:36.343794 18091 raft_consensus.cc:2781] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:33:36.343943 18091 raft_consensus.cc:482] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:33:36.344046 18091 raft_consensus.cc:3037] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:36.346634 18091 raft_consensus.cc:504] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.347406 18091 leader_election.cc:288] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [CANDIDATE]: Term 1 election: Requested vote from peers b18e97f766644c6a9c32307bdae3ff1e (127.0.96.193:44151), 960013f4ad2f42ef8081a99747a60b93 (127.0.96.194:38347)
I20260620 23:33:36.348492 17752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ead886dea240589898b34f6705620f" candidate_uuid: "7252a71932284b52b2852f5a61cc1909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18e97f766644c6a9c32307bdae3ff1e"
I20260620 23:33:36.348515 17886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ead886dea240589898b34f6705620f" candidate_uuid: "7252a71932284b52b2852f5a61cc1909" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "960013f4ad2f42ef8081a99747a60b93"
I20260620 23:33:36.348721 17752 raft_consensus.cc:3037] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:36.348809 17886 raft_consensus.cc:3037] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:36.352669 17752 raft_consensus.cc:2454] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7252a71932284b52b2852f5a61cc1909 in term 1.
I20260620 23:33:36.353291 17953 leader_election.cc:302] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7252a71932284b52b2852f5a61cc1909, b18e97f766644c6a9c32307bdae3ff1e; no voters:
I20260620 23:33:36.352690 17886 raft_consensus.cc:2454] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7252a71932284b52b2852f5a61cc1909 in term 1.
I20260620 23:33:36.353791 18091 raft_consensus.cc:2781] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:36.357471 18091 raft_consensus.cc:686] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 1 LEADER]: Becoming Leader. State: Replica: 7252a71932284b52b2852f5a61cc1909, State: Running, Role: LEADER
I20260620 23:33:36.358244 18091 consensus_queue.cc:237] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } }
I20260620 23:33:36.364559 17608 catalog_manager.cc:5495] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7252a71932284b52b2852f5a61cc1909 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "7252a71932284b52b2852f5a61cc1909" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7252a71932284b52b2852f5a61cc1909" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 40777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 } health_report { overall_health: UNKNOWN } } }
I20260620 23:33:36.399852 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.393419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.0.96.195:40777)
0620 23:33:36.393672 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.393682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.393774 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:33:36.394101 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.394106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.394216 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.394411 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:33:36.394857 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.395633 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.397233 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.397676 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.397825 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.398463 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:33:36.399147 (+ 684us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:33:36.399329 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.399402 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.399517 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:33:36.400023 18066 tablet.cc:2369] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:36.410136 17752 raft_consensus.cc:1261] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 1 FOLLOWER]: Refusing update from remote peer 7252a71932284b52b2852f5a61cc1909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:36.410305 17886 raft_consensus.cc:1261] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 1 FOLLOWER]: Refusing update from remote peer 7252a71932284b52b2852f5a61cc1909: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:36.411402 18091 consensus_queue.cc:1035] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18e97f766644c6a9c32307bdae3ff1e" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 44151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:36.411762 18094 consensus_queue.cc:1035] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [LEADER]: Connected to new peer: Peer: permanent_uuid: "960013f4ad2f42ef8081a99747a60b93" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 38347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:36.425319 18097 mvcc.cc:205] Tried to move back new op lower bound from 7299065513601986560 to 7299065513409929216. Current Snapshot: MvccSnapshot[applied={T|T < 7299065513601986560}]
I20260620 23:33:36.425621 18099 mvcc.cc:205] Tried to move back new op lower bound from 7299065513601986560 to 7299065513409929216. Current Snapshot: MvccSnapshot[applied={T|T < 7299065513601986560}]
I20260620 23:33:36.446086 17608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:36.446425 17608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:36.451349 17608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9455d3fd8de41a9af559fcc887c3670: Sending DeleteTablet for 3 replicas of tablet d2ead886dea240589898b34f6705620f
I20260620 23:33:36.453126 17732 tablet_service.cc:1514] Processing DeleteTablet for tablet d2ead886dea240589898b34f6705620f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:36 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:42284
I20260620 23:33:36.453099 17866 tablet_service.cc:1514] Processing DeleteTablet for tablet d2ead886dea240589898b34f6705620f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:36 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:45896
I20260620 23:33:36.453101 18000 tablet_service.cc:1514] Processing DeleteTablet for tablet d2ead886dea240589898b34f6705620f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:36 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:53836
I20260620 23:33:36.456173 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17669
I20260620 23:33:36.468844 18107 tablet_replica.cc:330] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: stopping tablet replica
I20260620 23:33:36.469622 18107 raft_consensus.cc:2229] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:36.469990 18109 tablet_replica.cc:330] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: stopping tablet replica
I20260620 23:33:36.470360 18107 raft_consensus.cc:2258] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:36.470534 18109 raft_consensus.cc:2229] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:33:36.471299 18109 raft_consensus.cc:2258] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:36.472193 18108 tablet_replica.cc:330] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: stopping tablet replica
I20260620 23:33:36.472631 18108 raft_consensus.cc:2229] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:36.473188 18108 raft_consensus.cc:2258] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:36.474025 18107 ts_tablet_manager.cc:1905] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:36.475515 18108 ts_tablet_manager.cc:1905] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:36.479315 18109 ts_tablet_manager.cc:1905] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 23:33:36 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:33:36.483395 18107 ts_tablet_manager.cc:1918] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:36.483888 18107 log.cc:1196] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/d2ead886dea240589898b34f6705620f
I20260620 23:33:36.484579 18107 ts_tablet_manager.cc:1939] T d2ead886dea240589898b34f6705620f P b18e97f766644c6a9c32307bdae3ff1e: Deleting consensus metadata
I20260620 23:33:36.487197 18109 ts_tablet_manager.cc:1918] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:36.487248 17592 catalog_manager.cc:4841] TS b18e97f766644c6a9c32307bdae3ff1e (127.0.96.193:44151): tablet d2ead886dea240589898b34f6705620f (table test-table [id=5a296d82d9e54de39a9cca5f87641176]) successfully deleted
I20260620 23:33:36.487422 18109 log.cc:1196] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/d2ead886dea240589898b34f6705620f
I20260620 23:33:36.487989 18109 ts_tablet_manager.cc:1939] T d2ead886dea240589898b34f6705620f P 7252a71932284b52b2852f5a61cc1909: Deleting consensus metadata
I20260620 23:33:36.490525 17593 catalog_manager.cc:4841] TS 7252a71932284b52b2852f5a61cc1909 (127.0.96.195:40777): tablet d2ead886dea240589898b34f6705620f (table test-table [id=5a296d82d9e54de39a9cca5f87641176]) successfully deleted
I20260620 23:33:36.490814 18108 ts_tablet_manager.cc:1918] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:36.491080 18108 log.cc:1196] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/d2ead886dea240589898b34f6705620f
I20260620 23:33:36.491714 18108 ts_tablet_manager.cc:1939] T d2ead886dea240589898b34f6705620f P 960013f4ad2f42ef8081a99747a60b93: Deleting consensus metadata
I20260620 23:33:36.494091 17592 catalog_manager.cc:4841] TS 960013f4ad2f42ef8081a99747a60b93 (127.0.96.194:38347): tablet d2ead886dea240589898b34f6705620f (table test-table [id=5a296d82d9e54de39a9cca5f87641176]) successfully deleted
I20260620 23:33:36.496879 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.458733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.96.193:44151)
0620 23:33:36.459055 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.459065 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.469197 (+ 10132us) server_negotiation.cc:407] Connection header received
0620 23:33:36.469453 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.469462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.469582 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.469744 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:36.472122 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.473292 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.475390 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.476121 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.476264 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.488021 (+ 11757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:36.488038 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:36.488062 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:36.488125 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:36.492975 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.494328 (+ 1353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.494340 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.494344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.494538 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.494997 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.495000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.495002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.495435 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:36.495708 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.496185 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.496442 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 23:33:36.497951 17792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:36.619037 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17669
I20260620 23:33:36.640970 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17803
Jun 20 23:33:36 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:33:36.662214 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.642098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.0.96.194:38347)
0620 23:33:36.642355 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.642364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.642389 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:33:36.642471 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.642481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.642552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.642670 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:36.644084 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.644963 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.647249 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.647729 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.647859 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.654335 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:36.654361 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:36.654384 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:36.654449 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:36.658134 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.659563 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.659578 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.659581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.659790 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.660209 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.660215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.660218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.660677 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:36.661062 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.661348 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.661779 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:33:36.663278 17926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:36.764525 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17803
I20260620 23:33:36.786513 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17937
Jun 20 23:33:36 dist-test-slave-v34r krb5kdc[17563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:33:36.808583 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:36.787496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.0.96.195:40777)
0620 23:33:36.787735 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:36.787743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:36.787767 (+ 24us) server_negotiation.cc:407] Connection header received
0620 23:33:36.788015 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:36.788024 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:36.788138 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:36.788299 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:36.789823 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.791068 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.792665 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:36.793425 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:36.793580 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:36.799468 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:36.799492 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:36.799516 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:36.799576 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:36.804204 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.805587 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.805604 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.805609 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.805797 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:36.806317 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:36.806323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:36.806331 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:36.806850 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:36.807183 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:36.807777 (+ 594us) server_negotiation.cc:299] Negotiation successful
0620 23:33:36.808119 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 23:33:36.810005 18060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:36.956537 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17937
I20260620 23:33:36.981174 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 17576
I20260620 23:33:36.982558 17638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:37.113521 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 17576
2026-06-20T23:33:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4654 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T23:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:37Z Disabled control of system clock
I20260620 23:33:37.207118 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:37921
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:37921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:33:37.547035 18118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:37.547353 18118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:37.547437 18118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:37.555559 18118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:37.555682 18118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:37.555722 18118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:37.555780 18118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:37.555831 18118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:37.565513 18118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:37921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:37.566839 18118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:37.568719 18118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:37.583401 18123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:37.583401 18124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:37.583469 18126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:37.584122 18118 server_base.cc:1034] running on GCE node
I20260620 23:33:37.585858 18118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:37.587724 18118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:37.589128 18118 hybrid_clock.cc:648] HybridClock initialized: now 1781998417589088 us; error 57 us; skew 500 ppm
I20260620 23:33:37.589577 18118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:37.592808 18118 webserver.cc:460] Webserver started at http://127.0.96.254:43259/ using document root <none> and password file <none>
I20260620 23:33:37.593962 18118 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:37.594235 18118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:37.594662 18118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:37.597993 18118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "9fc49625eb1943d4bb4102d4f76eac23"
format_stamp: "Formatted at 2026-06-20 23:33:37 on dist-test-slave-v34r"
server_key: "b2898805e4276f0957d4bafd897598bb"
server_key_iv: "601f59e97673f91d0adc077ada789a82"
server_key_version: "encryptionkey@0"
I20260620 23:33:37.598871 18118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "9fc49625eb1943d4bb4102d4f76eac23"
format_stamp: "Formatted at 2026-06-20 23:33:37 on dist-test-slave-v34r"
server_key: "b2898805e4276f0957d4bafd897598bb"
server_key_iv: "601f59e97673f91d0adc077ada789a82"
server_key_version: "encryptionkey@0"
I20260620 23:33:37.605173 18118 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 23:33:37.609606 18132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:37.611307 18118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:33:37.611529 18118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "9fc49625eb1943d4bb4102d4f76eac23"
format_stamp: "Formatted at 2026-06-20 23:33:37 on dist-test-slave-v34r"
server_key: "b2898805e4276f0957d4bafd897598bb"
server_key_iv: "601f59e97673f91d0adc077ada789a82"
server_key_version: "encryptionkey@0"
I20260620 23:33:37.611785 18118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:37.635448 18118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:37.636582 18118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:37.636909 18118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:37.659844 18118 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:37921
I20260620 23:33:37.659947 18183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:37921 every 8 connection(s)
I20260620 23:33:37.661393 18118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:37.663100 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18118
I20260620 23:33:37.663292 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:37.663761 387 external_mini_cluster.cc:1428] Setting key 98a3a22fce0d45237dfe90d7a35fb291
I20260620 23:33:37.667479 18184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:37.679205 18187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:37.666123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58994 (local address 127.0.96.254:37921)
0620 23:33:37.667401 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:37.667415 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:37.667479 (+ 64us) server_negotiation.cc:407] Connection header received
0620 23:33:37.669483 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:37.669510 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:37.670047 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:37.670487 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:37.671133 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:37.672539 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:37.673971 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:37.674694 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:37.674903 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:37.676454 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:37.676463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:37.676480 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:37.677174 (+ 694us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:37.677262 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:37.677295 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:37.677459 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:37.677731 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 23:33:37.678087 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":469,"threads_started":1}
I20260620 23:33:37.681898 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23: Bootstrap starting.
I20260620 23:33:37.686003 18184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:37.687465 18184 log.cc:824] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:37.691262 18184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23: No bootstrap required, opened a new log
I20260620 23:33:37.699654 18184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc49625eb1943d4bb4102d4f76eac23" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:33:37.700076 18184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:37.700155 18184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc49625eb1943d4bb4102d4f76eac23, State: Initialized, Role: FOLLOWER
I20260620 23:33:37.700778 18184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc49625eb1943d4bb4102d4f76eac23" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:33:37.701061 18184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:37.701237 18184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:37.701418 18184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:37.703985 18184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc49625eb1943d4bb4102d4f76eac23" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:33:37.704668 18184 leader_election.cc:302] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fc49625eb1943d4bb4102d4f76eac23; no voters:
I20260620 23:33:37.705476 18184 leader_election.cc:288] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:37.705783 18189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:37.706945 18189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [term 1 LEADER]: Becoming Leader. State: Replica: 9fc49625eb1943d4bb4102d4f76eac23, State: Running, Role: LEADER
I20260620 23:33:37.707676 18189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc49625eb1943d4bb4102d4f76eac23" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } }
I20260620 23:33:37.707854 18184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:37.712304 18191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fc49625eb1943d4bb4102d4f76eac23. Latest consensus state: current_term: 1 leader_uuid: "9fc49625eb1943d4bb4102d4f76eac23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc49625eb1943d4bb4102d4f76eac23" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } } }
I20260620 23:33:37.712255 18190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fc49625eb1943d4bb4102d4f76eac23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fc49625eb1943d4bb4102d4f76eac23" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37921 } } }
I20260620 23:33:37.713030 18191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:37.713325 18190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:37.722146 18198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:37.727273 18198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:37.736528 18198 catalog_manager.cc:1347] Generated new cluster ID: 61fb7cc159fa47d8ab70661415f3832e
I20260620 23:33:37.736678 18198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:37.757793 18198 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:37.759298 18198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:37.772634 18198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23: Generated new TSK 0
I20260620 23:33:37.773743 18198 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:33:37.785321 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:37921
--builtin_ntp_servers=127.0.96.212:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:33:38.144387 18208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:38.144727 18208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:38.144886 18208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:38.152494 18208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:38.152652 18208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:38.152833 18208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:38.162457 18208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:37921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:38.163802 18208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:38.165712 18208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:38.180912 18214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:38.181001 18216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:38.180881 18213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:38.183203 18208 server_base.cc:1034] running on GCE node
I20260620 23:33:38.183929 18208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:38.185129 18208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:38.186394 18208 hybrid_clock.cc:648] HybridClock initialized: now 1781998418186326 us; error 62 us; skew 500 ppm
I20260620 23:33:38.186756 18208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:38.189675 18208 webserver.cc:460] Webserver started at http://127.0.96.193:35701/ using document root <none> and password file <none>
I20260620 23:33:38.190490 18208 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:38.190627 18208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:38.190959 18208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:38.194073 18208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "8e48a0f8265743d98934df164b86da5d"
format_stamp: "Formatted at 2026-06-20 23:33:38 on dist-test-slave-v34r"
server_key: "dee7bb36d6a16af609922c1a0a81195d"
server_key_iv: "e7f6a052b5085ca47abc56b0f7c42742"
server_key_version: "encryptionkey@0"
I20260620 23:33:38.194881 18208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "8e48a0f8265743d98934df164b86da5d"
format_stamp: "Formatted at 2026-06-20 23:33:38 on dist-test-slave-v34r"
server_key: "dee7bb36d6a16af609922c1a0a81195d"
server_key_iv: "e7f6a052b5085ca47abc56b0f7c42742"
server_key_version: "encryptionkey@0"
I20260620 23:33:38.200670 18208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 23:33:38.204926 18222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:38.206768 18208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:38.207010 18208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "8e48a0f8265743d98934df164b86da5d"
format_stamp: "Formatted at 2026-06-20 23:33:38 on dist-test-slave-v34r"
server_key: "dee7bb36d6a16af609922c1a0a81195d"
server_key_iv: "e7f6a052b5085ca47abc56b0f7c42742"
server_key_version: "encryptionkey@0"
I20260620 23:33:38.207305 18208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:38.237268 18208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:38.238200 18208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:38.238533 18208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:38.239707 18208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:38.241559 18208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:38.241947 18208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:38.242106 18208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:38.242208 18208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:38.282429 18208 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:43657
I20260620 23:33:38.282593 18334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:43657 every 8 connection(s)
I20260620 23:33:38.283811 18208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:38.292554 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18208
I20260620 23:33:38.292789 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:33:38.293313 387 external_mini_cluster.cc:1428] Setting key f4cd911cfc8b40dc23b8063020ab3377
I20260620 23:33:38.296684 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:37921
--builtin_ntp_servers=127.0.96.212:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:33:38.300869 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:38.287315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:33325 (local address 127.0.96.254:37921)
0620 23:33:38.288091 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:38.288097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:38.289519 (+ 1422us) server_negotiation.cc:407] Connection header received
0620 23:33:38.291840 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:38.291848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:38.291969 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:38.292117 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:38.293865 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.294868 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.296877 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.297713 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.297922 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:38.299579 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:38.299589 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:38.299598 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:38.299633 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:38.299706 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:38.299729 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:38.299863 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:38.300193 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 23:33:38.300347 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":453,"threads_started":1}
I20260620 23:33:38.301733 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:38.287881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37921 (local address 127.0.96.193:33325)
0620 23:33:38.289297 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:38.289357 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:38.291410 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:38.292471 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:38.292482 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:38.292638 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:38.293635 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:38.293663 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.295148 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.295154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:38.296671 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:38.296688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.298031 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.298037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:38.298262 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:38.299173 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:38.299176 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:38.299224 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:38.299314 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:38.299903 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:38.299913 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:38.299920 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 23:33:38.300144 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 23:33:38.300509 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":369,"threads_started":1}
I20260620 23:33:38.303562 18335 heartbeater.cc:347] Connected to a master server at 127.0.96.254:37921
I20260620 23:33:38.303913 18335 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:38.304715 18335 heartbeater.cc:511] Master 127.0.96.254:37921 requested a full tablet report, sending...
I20260620 23:33:38.306885 18149 ts_manager.cc:194] Registered new tserver with Master: 8e48a0f8265743d98934df164b86da5d (127.0.96.193:43657)
I20260620 23:33:38.309063 18149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.193:33325
W20260620 23:33:38.723114 18339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:38.723508 18339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:38.723681 18339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:38.731325 18339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:38.731487 18339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:38.731665 18339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:38.741405 18339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:37921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:38.742848 18339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:38.744769 18339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:38.759454 18345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:38.759508 18347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:38.759385 18344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:38.761057 18339 server_base.cc:1034] running on GCE node
I20260620 23:33:38.762017 18339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:38.763391 18339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:38.764704 18339 hybrid_clock.cc:648] HybridClock initialized: now 1781998418764642 us; error 66 us; skew 500 ppm
I20260620 23:33:38.765239 18339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:38.768327 18339 webserver.cc:460] Webserver started at http://127.0.96.194:46399/ using document root <none> and password file <none>
I20260620 23:33:38.769366 18339 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:38.769531 18339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:38.769929 18339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:38.773902 18339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "79fc7176f76a4fbab1c1a04634474b4a"
format_stamp: "Formatted at 2026-06-20 23:33:38 on dist-test-slave-v34r"
server_key: "8fa56a341facc05e302d62454321dd25"
server_key_iv: "731ab91c02fe4075ac8528478a56f498"
server_key_version: "encryptionkey@0"
I20260620 23:33:38.774964 18339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "79fc7176f76a4fbab1c1a04634474b4a"
format_stamp: "Formatted at 2026-06-20 23:33:38 on dist-test-slave-v34r"
server_key: "8fa56a341facc05e302d62454321dd25"
server_key_iv: "731ab91c02fe4075ac8528478a56f498"
server_key_version: "encryptionkey@0"
I20260620 23:33:38.782087 18339 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260620 23:33:38.787092 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:38.788869 18339 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:33:38.789237 18339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "79fc7176f76a4fbab1c1a04634474b4a"
format_stamp: "Formatted at 2026-06-20 23:33:38 on dist-test-slave-v34r"
server_key: "8fa56a341facc05e302d62454321dd25"
server_key_iv: "731ab91c02fe4075ac8528478a56f498"
server_key_version: "encryptionkey@0"
I20260620 23:33:38.789534 18339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:38.806317 18339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:38.807192 18339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:38.807500 18339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:38.808743 18339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:38.810482 18339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:38.810576 18339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:38.810659 18339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:38.810724 18339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:38.851277 18339 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:44355
I20260620 23:33:38.851321 18465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:44355 every 8 connection(s)
I20260620 23:33:38.852648 18339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:38.854429 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18339
I20260620 23:33:38.854609 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:33:38.855084 387 external_mini_cluster.cc:1428] Setting key a58f401e3586ea741a07486f690bf70f
I20260620 23:33:38.859194 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:37921
--builtin_ntp_servers=127.0.96.212:42129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:33:38.871907 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:38.856383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:36505 (local address 127.0.96.254:37921)
0620 23:33:38.856905 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:38.856910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:38.859071 (+ 2161us) server_negotiation.cc:407] Connection header received
0620 23:33:38.862222 (+ 3151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:38.862229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:38.862347 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:38.862491 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:38.864156 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.865154 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.867843 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.868593 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.868764 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:38.870862 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:38.870868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:38.870872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:38.870898 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:38.870949 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:38.870963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:38.871073 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:38.871343 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 23:33:38.871491 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":276,"threads_started":1}
I20260620 23:33:38.873405 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:38.857247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37921 (local address 127.0.96.194:36505)
0620 23:33:38.858833 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:38.858902 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:38.861821 (+ 2919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:38.862835 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:38.862850 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:38.862987 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:38.863872 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:38.863900 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.865361 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.865370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:38.867618 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:38.867639 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:38.869141 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:38.869150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:38.869446 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:38.870389 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:38.870392 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:38.870450 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:38.870570 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:38.871073 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:38.871088 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:38.871099 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:33:38.871333 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:33:38.871920 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":492,"threads_started":1}
I20260620 23:33:38.875442 18466 heartbeater.cc:347] Connected to a master server at 127.0.96.254:37921
I20260620 23:33:38.875859 18466 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:38.876888 18466 heartbeater.cc:511] Master 127.0.96.254:37921 requested a full tablet report, sending...
I20260620 23:33:38.879056 18149 ts_manager.cc:194] Registered new tserver with Master: 79fc7176f76a4fbab1c1a04634474b4a (127.0.96.194:44355)
I20260620 23:33:38.880447 18149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.194:36505
W20260620 23:33:39.214118 18470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:39.214408 18470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:39.214497 18470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:39.222025 18470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:39.222147 18470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:39.222255 18470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:39.231880 18470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:42129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:37921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:39.233229 18470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:39.234946 18470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:39.250082 18476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:39.250193 18478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:39.250707 18475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:39.250936 18470 server_base.cc:1034] running on GCE node
I20260620 23:33:39.251637 18470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:39.252653 18470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:39.253888 18470 hybrid_clock.cc:648] HybridClock initialized: now 1781998419253814 us; error 69 us; skew 500 ppm
I20260620 23:33:39.254268 18470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:39.257053 18470 webserver.cc:460] Webserver started at http://127.0.96.195:40753/ using document root <none> and password file <none>
I20260620 23:33:39.257839 18470 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:39.257953 18470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:39.258240 18470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:39.261382 18470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "3822377638e3493dad146c93725377ea"
format_stamp: "Formatted at 2026-06-20 23:33:39 on dist-test-slave-v34r"
server_key: "53383d065d01fe87c69e0ae96438afe5"
server_key_iv: "aeb9338fcbe9934b73af388f247c70db"
server_key_version: "encryptionkey@0"
I20260620 23:33:39.262146 18470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "3822377638e3493dad146c93725377ea"
format_stamp: "Formatted at 2026-06-20 23:33:39 on dist-test-slave-v34r"
server_key: "53383d065d01fe87c69e0ae96438afe5"
server_key_iv: "aeb9338fcbe9934b73af388f247c70db"
server_key_version: "encryptionkey@0"
I20260620 23:33:39.267794 18470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 23:33:39.271811 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:39.273589 18470 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:39.273797 18470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "3822377638e3493dad146c93725377ea"
format_stamp: "Formatted at 2026-06-20 23:33:39 on dist-test-slave-v34r"
server_key: "53383d065d01fe87c69e0ae96438afe5"
server_key_iv: "aeb9338fcbe9934b73af388f247c70db"
server_key_version: "encryptionkey@0"
I20260620 23:33:39.274016 18470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:39.301443 18470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:39.302340 18470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:39.302673 18470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:39.303786 18470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:39.305423 18470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:39.305518 18470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:39.305686 18470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:39.305787 18470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:39.312523 18335 heartbeater.cc:503] Master 127.0.96.254:37921 was elected leader, sending a full tablet report...
I20260620 23:33:39.345901 18470 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:41465
I20260620 23:33:39.346170 18596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:41465 every 8 connection(s)
I20260620 23:33:39.347319 18470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:39.351650 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18470
I20260620 23:33:39.351878 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:33:39.352573 387 external_mini_cluster.cc:1428] Setting key 7912172c772bd4adecb420c34e1285cf
I20260620 23:33:39.363086 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.351003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:58311 (local address 127.0.96.254:37921)
0620 23:33:39.351368 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.351378 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.353161 (+ 1783us) server_negotiation.cc:407] Connection header received
0620 23:33:39.355589 (+ 2428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.355594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.355685 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.355786 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:39.357158 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.357969 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.359751 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.360279 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.360440 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.362161 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:39.362167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:39.362171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:39.362208 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:39.362258 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:39.362274 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:39.362378 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.362684 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.362810 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260620 23:33:39.364064 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.351648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37921 (local address 127.0.96.195:58311)
0620 23:33:39.352941 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:39.353022 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:39.355218 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:39.356090 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:39.356103 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:39.356247 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:39.356892 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.356909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.358194 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.358203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.359509 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.359522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.360586 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.360592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.360814 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.361772 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:39.361774 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:39.361816 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:39.361910 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:39.362389 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:39.362396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:39.362402 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 23:33:39.362570 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 23:33:39.362897 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":476,"threads_started":1}
I20260620 23:33:39.366163 18597 heartbeater.cc:347] Connected to a master server at 127.0.96.254:37921
I20260620 23:33:39.366521 18597 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:39.367420 18597 heartbeater.cc:511] Master 127.0.96.254:37921 requested a full tablet report, sending...
I20260620 23:33:39.369693 18149 ts_manager.cc:194] Registered new tserver with Master: 3822377638e3493dad146c93725377ea (127.0.96.195:41465)
I20260620 23:33:39.370972 18149 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.195:58311
I20260620 23:33:39.371111 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:33:39.382953 18468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.376394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58998 (local address 127.0.96.254:37921)
0620 23:33:39.376630 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.376639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.377171 (+ 532us) server_negotiation.cc:407] Connection header received
0620 23:33:39.377400 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.377406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.377497 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.377610 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:39.378104 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.378845 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.380324 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.380765 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.380917 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.382193 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:39.382198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:39.382202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:39.382221 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:39.382254 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:39.382267 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:39.382370 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.382514 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.382614 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:33:39.388193 18149 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:33:39.397107 18149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:33:39.428385 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.413380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:41465 (local address 127.0.0.1:55356)
0620 23:33:39.414376 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:39.414433 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:39.414659 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:39.416073 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:39.416087 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:39.416137 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:39.416804 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.416821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.419861 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.419872 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.423529 (+ 3657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.423563 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.427050 (+ 3487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.427061 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.427463 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.427537 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 23:33:39.427713 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 23:33:39.427968 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":350,"threads_started":1}
I20260620 23:33:39.429769 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.413752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.0.96.195:41465)
0620 23:33:39.414596 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.414607 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.414648 (+ 41us) server_negotiation.cc:407] Connection header received
0620 23:33:39.414921 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.414930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.415396 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.415618 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:39.417141 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.419581 (+ 2440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.423938 (+ 4357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.426117 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.426360 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.426821 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.429137 (+ 2316us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.429390 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":944,"threads_started":1}
I20260620 23:33:39.435793 18532 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7a2f3026324d3487a62a3e6114dc63 (DEFAULT_TABLE table=test-table [id=f4b127ce109344a7b89670b9003f47c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:39.437925 18532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7a2f3026324d3487a62a3e6114dc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:39.438199 18609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.414356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:44355 (local address 127.0.0.1:41244)
0620 23:33:39.428723 (+ 14367us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:39.428747 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:39.428883 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:39.430518 (+ 1635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:39.430522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:39.430548 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:39.430713 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.430721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.432644 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.432650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.434725 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.434740 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.437098 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.437105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.437445 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.437491 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 23:33:39.437601 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 23:33:39.437781 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14267,"mutex_wait_us":1,"thread_start_us":17877,"threads_started":1}
I20260620 23:33:39.438431 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.415307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.96.194:44355)
0620 23:33:39.424427 (+ 9120us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.424439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.429120 (+ 4681us) server_negotiation.cc:407] Connection header received
0620 23:33:39.429276 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.429286 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.429649 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.430281 (+ 632us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:39.430879 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.432466 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.434936 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.436436 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.436632 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.436919 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.437774 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.438032 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8892,"thread_start_us":8709,"threads_started":1}
I20260620 23:33:39.440442 18612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.414687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38102)
0620 23:33:39.431751 (+ 17064us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:39.431778 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:39.431955 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:39.433439 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:39.433443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:39.433470 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:39.433910 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.433922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.435507 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.435515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.437435 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.437448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.439665 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.439672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.439796 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.439822 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 23:33:39.439930 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:33:39.440066 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16855,"mutex_wait_us":3,"thread_start_us":16095,"threads_started":1}
I20260620 23:33:39.440497 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.415391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.0.96.193:43657)
0620 23:33:39.428000 (+ 12609us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.428011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.432478 (+ 4467us) server_negotiation.cc:407] Connection header received
0620 23:33:39.432629 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.432639 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.432957 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.433180 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:39.434110 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.435379 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.437640 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.439074 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.439205 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.439523 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.439981 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.440146 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12366,"thread_start_us":12628,"threads_started":1}
I20260620 23:33:39.443234 18401 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7a2f3026324d3487a62a3e6114dc63 (DEFAULT_TABLE table=test-table [id=f4b127ce109344a7b89670b9003f47c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:39.444360 18270 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7a2f3026324d3487a62a3e6114dc63 (DEFAULT_TABLE table=test-table [id=f4b127ce109344a7b89670b9003f47c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:39.445168 18401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7a2f3026324d3487a62a3e6114dc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:39.445746 18270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7a2f3026324d3487a62a3e6114dc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:39.456939 18615 tablet_bootstrap.cc:492] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Bootstrap starting.
I20260620 23:33:39.459494 18616 tablet_bootstrap.cc:492] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: Bootstrap starting.
I20260620 23:33:39.461115 18615 tablet_bootstrap.cc:654] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:39.462345 18617 tablet_bootstrap.cc:492] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Bootstrap starting.
I20260620 23:33:39.463711 18615 log.cc:824] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:39.463975 18616 tablet_bootstrap.cc:654] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:39.465422 18616 log.cc:824] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:39.465845 18617 tablet_bootstrap.cc:654] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:39.467023 18617 log.cc:824] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:39.467726 18615 tablet_bootstrap.cc:492] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: No bootstrap required, opened a new log
I20260620 23:33:39.468082 18615 ts_tablet_manager.cc:1397] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260620 23:33:39.468647 18616 tablet_bootstrap.cc:492] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: No bootstrap required, opened a new log
I20260620 23:33:39.468950 18616 ts_tablet_manager.cc:1397] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 23:33:39.469822 18617 tablet_bootstrap.cc:492] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: No bootstrap required, opened a new log
I20260620 23:33:39.470090 18617 ts_tablet_manager.cc:1397] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260620 23:33:39.478416 18617 raft_consensus.cc:348] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.478854 18617 raft_consensus.cc:374] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:39.478957 18617 raft_consensus.cc:729] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79fc7176f76a4fbab1c1a04634474b4a, State: Initialized, Role: FOLLOWER
I20260620 23:33:39.479558 18617 consensus_queue.cc:260] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.480238 18615 raft_consensus.cc:348] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.480674 18466 heartbeater.cc:503] Master 127.0.96.254:37921 was elected leader, sending a full tablet report...
I20260620 23:33:39.480796 18615 raft_consensus.cc:374] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:39.480899 18615 raft_consensus.cc:729] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3822377638e3493dad146c93725377ea, State: Initialized, Role: FOLLOWER
I20260620 23:33:39.481379 18617 ts_tablet_manager.cc:1428] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260620 23:33:39.481479 18616 raft_consensus.cc:348] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.481950 18616 raft_consensus.cc:374] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:39.481787 18615 consensus_queue.cc:260] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.482060 18616 raft_consensus.cc:729] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e48a0f8265743d98934df164b86da5d, State: Initialized, Role: FOLLOWER
I20260620 23:33:39.482774 18616 consensus_queue.cc:260] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.484006 18615 ts_tablet_manager.cc:1428] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260620 23:33:39.484617 18616 ts_tablet_manager.cc:1428] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260620 23:33:39.484812 18597 heartbeater.cc:503] Master 127.0.96.254:37921 was elected leader, sending a full tablet report...
W20260620 23:33:39.537098 18336 tablet.cc:2369] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:39.599243 18598 tablet.cc:2369] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:39.605199 18467 tablet.cc:2369] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:39.756951 18621 raft_consensus.cc:482] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:33:39.757398 18621 raft_consensus.cc:504] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.759142 18621 leader_election.cc:288] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3822377638e3493dad146c93725377ea (127.0.96.195:41465), 8e48a0f8265743d98934df164b86da5d (127.0.96.193:43657)
I20260620 23:33:39.768115 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.759797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:54619 (local address 127.0.96.195:41465)
0620 23:33:39.760041 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.760046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.760380 (+ 334us) server_negotiation.cc:407] Connection header received
0620 23:33:39.760584 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.760593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.760681 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.760852 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:39.761643 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.762797 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.766195 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.767050 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.767264 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.767347 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.767684 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.767822 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 23:33:39.768224 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.759505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:41465 (local address 127.0.96.194:54619)
0620 23:33:39.760210 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:39.760233 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:39.760390 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:39.761027 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:39.761035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:39.761067 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:39.761460 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.761475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.762974 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.763019 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.765835 (+ 2816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.765851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.767273 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.767280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.767501 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.767528 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:33:39.767641 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:33:39.767804 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":155,"thread_start_us":371,"threads_started":1}
I20260620 23:33:39.768275 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.760701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:43013 (local address 127.0.96.193:43657)
0620 23:33:39.760969 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.761003 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.761069 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:33:39.761274 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.761278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.761376 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.761463 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:39.761996 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.763029 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.766168 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.767093 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.767377 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.767481 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.767822 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.767980 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:33:39.768697 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.760362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.96.194:43013)
0620 23:33:39.760949 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:39.760966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:39.761117 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:39.761515 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:39.761522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:39.761549 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:39.761836 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.761843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.763184 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.763189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.765834 (+ 2645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:39.765851 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.767418 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.767426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:39.767621 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.767650 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:33:39.767784 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:33:39.767964 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":116,"spinlock_wait_cycles":768,"thread_start_us":419,"threads_started":1}
I20260620 23:33:39.770069 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a2f3026324d3487a62a3e6114dc63" candidate_uuid: "79fc7176f76a4fbab1c1a04634474b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3822377638e3493dad146c93725377ea" is_pre_election: true
I20260620 23:33:39.770069 18290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a2f3026324d3487a62a3e6114dc63" candidate_uuid: "79fc7176f76a4fbab1c1a04634474b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e48a0f8265743d98934df164b86da5d" is_pre_election: true
I20260620 23:33:39.770843 18552 raft_consensus.cc:2454] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79fc7176f76a4fbab1c1a04634474b4a in term 0.
I20260620 23:33:39.770828 18290 raft_consensus.cc:2454] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79fc7176f76a4fbab1c1a04634474b4a in term 0.
I20260620 23:33:39.771679 18354 leader_election.cc:302] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79fc7176f76a4fbab1c1a04634474b4a, 8e48a0f8265743d98934df164b86da5d; no voters:
I20260620 23:33:39.772138 18621 raft_consensus.cc:2781] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:33:39.772347 18621 raft_consensus.cc:482] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:33:39.772459 18621 raft_consensus.cc:3037] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:39.775105 18621 raft_consensus.cc:504] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.775787 18621 leader_election.cc:288] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [CANDIDATE]: Term 1 election: Requested vote from peers 3822377638e3493dad146c93725377ea (127.0.96.195:41465), 8e48a0f8265743d98934df164b86da5d (127.0.96.193:43657)
I20260620 23:33:39.776559 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a2f3026324d3487a62a3e6114dc63" candidate_uuid: "79fc7176f76a4fbab1c1a04634474b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3822377638e3493dad146c93725377ea"
I20260620 23:33:39.776611 18290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7a2f3026324d3487a62a3e6114dc63" candidate_uuid: "79fc7176f76a4fbab1c1a04634474b4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e48a0f8265743d98934df164b86da5d"
I20260620 23:33:39.776866 18552 raft_consensus.cc:3037] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:39.776866 18290 raft_consensus.cc:3037] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:39.780682 18290 raft_consensus.cc:2454] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79fc7176f76a4fbab1c1a04634474b4a in term 1.
I20260620 23:33:39.780827 18552 raft_consensus.cc:2454] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79fc7176f76a4fbab1c1a04634474b4a in term 1.
I20260620 23:33:39.781389 18354 leader_election.cc:302] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3822377638e3493dad146c93725377ea, 79fc7176f76a4fbab1c1a04634474b4a; no voters:
I20260620 23:33:39.781788 18621 raft_consensus.cc:2781] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:39.782305 18621 raft_consensus.cc:686] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 1 LEADER]: Becoming Leader. State: Replica: 79fc7176f76a4fbab1c1a04634474b4a, State: Running, Role: LEADER
I20260620 23:33:39.782876 18621 consensus_queue.cc:237] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } }
I20260620 23:33:39.787992 18148 catalog_manager.cc:5495] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 79fc7176f76a4fbab1c1a04634474b4a (127.0.96.194). New cstate: current_term: 1 leader_uuid: "79fc7176f76a4fbab1c1a04634474b4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79fc7176f76a4fbab1c1a04634474b4a" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 } health_report { overall_health: UNKNOWN } } }
I20260620 23:33:39.882735 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.877341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.0.96.194:44355)
0620 23:33:39.877615 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.877624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.877659 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:33:39.877754 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.877757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.877853 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.878008 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:33:39.878357 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.879141 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.880666 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.881140 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.881241 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.881728 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:33:39.882232 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:33:39.882344 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.882452 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.882514 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 23:33:39.893309 18290 raft_consensus.cc:1261] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 1 FOLLOWER]: Refusing update from remote peer 79fc7176f76a4fbab1c1a04634474b4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:39.893309 18552 raft_consensus.cc:1261] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 1 FOLLOWER]: Refusing update from remote peer 79fc7176f76a4fbab1c1a04634474b4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:39.894457 18626 consensus_queue.cc:1035] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e48a0f8265743d98934df164b86da5d" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:39.894928 18621 consensus_queue.cc:1035] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3822377638e3493dad146c93725377ea" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:39.909221 18631 mvcc.cc:205] Tried to move back new op lower bound from 7299065527868731392 to 7299065527434633216. Current Snapshot: MvccSnapshot[applied={T|T < 7299065527868731392}]
I20260620 23:33:39.911545 18630 mvcc.cc:205] Tried to move back new op lower bound from 7299065527868731392 to 7299065527434633216. Current Snapshot: MvccSnapshot[applied={T|T < 7299065527868731392}]
I20260620 23:33:39.925760 18148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:39.926038 18148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:39.930819 18148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fc49625eb1943d4bb4102d4f76eac23: Sending DeleteTablet for 3 replicas of tablet 5a7a2f3026324d3487a62a3e6114dc63
I20260620 23:33:39.932515 18401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7a2f3026324d3487a62a3e6114dc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:39 UTC) from {username='slave'} at 127.0.0.1:41244
I20260620 23:33:39.932801 18532 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7a2f3026324d3487a62a3e6114dc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:39 UTC) from {username='slave'} at 127.0.0.1:55356
I20260620 23:33:39.933209 18270 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7a2f3026324d3487a62a3e6114dc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:39 UTC) from {username='slave'} at 127.0.0.1:38102
I20260620 23:33:39.933807 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18208
I20260620 23:33:39.950533 18640 tablet_replica.cc:330] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: stopping tablet replica
I20260620 23:33:39.951398 18640 raft_consensus.cc:2229] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:33:39.952519 18640 raft_consensus.cc:2258] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:39.956897 18640 ts_tablet_manager.cc:1905] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:39.961844 18641 tablet_replica.cc:330] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d: stopping tablet replica
I20260620 23:33:39.962529 18641 raft_consensus.cc:2229] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:39.963534 18639 tablet_replica.cc:330] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: stopping tablet replica
I20260620 23:33:39.963673 18641 raft_consensus.cc:2258] T 5a7a2f3026324d3487a62a3e6114dc63 P 8e48a0f8265743d98934df164b86da5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:39.965492 18639 raft_consensus.cc:2229] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:39.966197 18639 raft_consensus.cc:2258] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:39.967177 18614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:39.935765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.0.96.193:43657)
0620 23:33:39.936033 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:39.936043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:39.947040 (+ 10997us) server_negotiation.cc:407] Connection header received
0620 23:33:39.947834 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:39.947845 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:39.948038 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:39.948156 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:39.949281 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.950610 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.952819 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:39.953610 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:39.953739 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:39.964319 (+ 10580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:39.964336 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:39.964358 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:39.965536 (+ 1178us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:39.965630 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:39.965675 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:39.965880 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:39.966553 (+ 673us) server_negotiation.cc:299] Negotiation successful
0620 23:33:39.966736 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:33:39.968681 18640 ts_tablet_manager.cc:1918] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:39.968798 18330 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:39.968895 18640 log.cc:1196] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/5a7a2f3026324d3487a62a3e6114dc63
I20260620 23:33:39.969072 18639 ts_tablet_manager.cc:1905] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:39.969970 18640 ts_tablet_manager.cc:1939] T 5a7a2f3026324d3487a62a3e6114dc63 P 79fc7176f76a4fbab1c1a04634474b4a: Deleting consensus metadata
I20260620 23:33:39.973347 18136 catalog_manager.cc:4841] TS 79fc7176f76a4fbab1c1a04634474b4a (127.0.96.194:44355): tablet 5a7a2f3026324d3487a62a3e6114dc63 (table test-table [id=f4b127ce109344a7b89670b9003f47c5]) successfully deleted
I20260620 23:33:39.978590 18639 ts_tablet_manager.cc:1918] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:39.978950 18639 log.cc:1196] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/5a7a2f3026324d3487a62a3e6114dc63
I20260620 23:33:39.979512 18639 ts_tablet_manager.cc:1939] T 5a7a2f3026324d3487a62a3e6114dc63 P 3822377638e3493dad146c93725377ea: Deleting consensus metadata
I20260620 23:33:39.981848 18133 catalog_manager.cc:4841] TS 3822377638e3493dad146c93725377ea (127.0.96.195:41465): tablet 5a7a2f3026324d3487a62a3e6114dc63 (table test-table [id=f4b127ce109344a7b89670b9003f47c5]) successfully deleted
I20260620 23:33:40.082201 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18208
I20260620 23:33:40.104215 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18339
W20260620 23:33:40.104147 18133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:43657 (error 108)
I20260620 23:33:40.107028 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.105854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38120)
0620 23:33:40.106648 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.106801 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":505,"threads_started":1}
W20260620 23:33:40.107432 18133 catalog_manager.cc:4584] TS 8e48a0f8265743d98934df164b86da5d (127.0.96.193:43657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a7a2f3026324d3487a62a3e6114dc63: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
I20260620 23:33:40.114252 18613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.105353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.0.96.194:44355)
0620 23:33:40.105549 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:40.105555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:40.105578 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:33:40.105757 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:40.105768 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:40.105902 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:40.106038 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:40.106655 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:40.107903 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:40.110085 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:40.110846 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:40.111020 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:40.112426 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:40.112437 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:40.112458 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:40.113325 (+ 867us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:40.113408 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:40.113435 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:40.113552 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:40.113719 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 23:33:40.113840 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:33:40.115289 18461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:40.161597 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.160900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38130)
0620 23:33:40.161252 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.161387 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260620 23:33:40.163120 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.162600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38144)
0620 23:33:40.162809 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.162914 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:33:40.224197 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.223468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38148)
0620 23:33:40.223824 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.223971 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 23:33:40.225687 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.225139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38156)
0620 23:33:40.225377 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.225489 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 23:33:40.235728 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18339
I20260620 23:33:40.256693 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18470
I20260620 23:33:40.264859 18610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.257736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.96.195:41465)
0620 23:33:40.258005 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:40.258010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:40.258037 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:33:40.258191 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:40.258196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:40.258309 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:40.258426 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:40.258912 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:40.259736 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:40.261301 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:40.261790 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:40.261927 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:40.263076 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:40.263087 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:40.263107 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:40.263893 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:40.263959 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:40.263984 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:40.264097 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:40.264260 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:33:40.264436 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:33:40.265902 18592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:40.306870 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.306297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38160)
0620 23:33:40.306554 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.306664 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:33:40.308569 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.307910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38174)
0620 23:33:40.308207 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.308318 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260620 23:33:40.360455 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18470
I20260620 23:33:40.378212 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18118
I20260620 23:33:40.379612 18179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:40.507640 18645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:40.506862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:43657 (local address 127.0.0.1:38178)
0620 23:33:40.507083 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:40.507263 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 23:33:40.508095 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18118
2026-06-20T23:33:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3388 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:33:40 dist-test-slave-v34r krb5kdc[18651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:33:40 dist-test-slave-v34r krb5kdc[18651](info): set up 2 sockets
Jun 20 23:33:40 dist-test-slave-v34r krb5kdc[18651](info): commencing operation
krb5kdc: starting...
W20260620 23:33:41.649420 387 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.007s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:33:41 dist-test-slave-v34r krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:33:41.933053 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:33121
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:33121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:42.282703 18664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:42.283025 18664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:42.283212 18664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:42.290830 18664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:42.290969 18664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:42.291016 18664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:42.291154 18664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:42.291260 18664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:42.302899 18664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:33121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:33121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:42.304656 18664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:42.306143 18664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:42.319880 18669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:42.319818 18670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:42.319761 18672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:42.321166 18664 server_base.cc:1034] running on GCE node
I20260620 23:33:42.322039 18664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:42.323664 18664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:42.324937 18664 hybrid_clock.cc:648] HybridClock initialized: now 1781998422324882 us; error 49 us; skew 500 ppm
I20260620 23:33:42.325435 18664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:42 dist-test-slave-v34r krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:42.334721 18664 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:33:42.336581 18664 webserver.cc:460] Webserver started at http://127.0.96.254:38997/ using document root <none> and password file <none>
I20260620 23:33:42.337442 18664 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:42.337595 18664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:42.337915 18664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:42.340899 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "ae88022bf88f4fa78a029138b990cacb"
format_stamp: "Formatted at 2026-06-20 23:33:42 on dist-test-slave-v34r"
server_key: "5233f34063d6b42144b6b5e56137a5da"
server_key_iv: "474c5b7ff22b8eb04319daec72442753"
server_key_version: "encryptionkey@0"
I20260620 23:33:42.341714 18664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "ae88022bf88f4fa78a029138b990cacb"
format_stamp: "Formatted at 2026-06-20 23:33:42 on dist-test-slave-v34r"
server_key: "5233f34063d6b42144b6b5e56137a5da"
server_key_iv: "474c5b7ff22b8eb04319daec72442753"
server_key_version: "encryptionkey@0"
I20260620 23:33:42.347183 18664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:33:42.351042 18679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:42.352604 18664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 23:33:42.352826 18664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "ae88022bf88f4fa78a029138b990cacb"
format_stamp: "Formatted at 2026-06-20 23:33:42 on dist-test-slave-v34r"
server_key: "5233f34063d6b42144b6b5e56137a5da"
server_key_iv: "474c5b7ff22b8eb04319daec72442753"
server_key_version: "encryptionkey@0"
I20260620 23:33:42.353101 18664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:42.383891 18664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:42.388564 18664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:42.388857 18664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:42.409871 18664 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:33121
I20260620 23:33:42.409968 18730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:33121 every 8 connection(s)
I20260620 23:33:42.411321 18664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:42.415021 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18664
I20260620 23:33:42.415246 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:42.415858 387 external_mini_cluster.cc:1428] Setting key 7819d96a49fc9e0b6e9c9fcf4b1d8ff0
I20260620 23:33:42.416712 18731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:33:42 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:42.431617 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb: Bootstrap starting.
I20260620 23:33:42.435956 18731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:42.437304 18731 log.cc:824] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:42.439587 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:42.418323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.0.96.254:33121)
0620 23:33:42.419300 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:42.419314 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:42.419364 (+ 50us) server_negotiation.cc:407] Connection header received
0620 23:33:42.421082 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:42.421105 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:42.421547 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:42.421963 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:42.423506 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:42.424967 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:42.426387 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:42.427086 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:42.427318 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:42.431652 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:42.431660 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:42.431677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:42.431715 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:42.435215 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:42.435916 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:42.435925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:42.435932 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:42.436150 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:42.436538 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:42.436544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:42.436547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:42.437214 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:42.437529 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:42.438032 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 23:33:42.438529 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":400,"threads_started":1}
I20260620 23:33:42.441452 18731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb: No bootstrap required, opened a new log
I20260620 23:33:42.450155 18731 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88022bf88f4fa78a029138b990cacb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 33121 } }
I20260620 23:33:42.450722 18731 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:42.450862 18731 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae88022bf88f4fa78a029138b990cacb, State: Initialized, Role: FOLLOWER
I20260620 23:33:42.451608 18731 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88022bf88f4fa78a029138b990cacb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 33121 } }
I20260620 23:33:42.451913 18731 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:42.452077 18731 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:42.452284 18731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:42.455085 18731 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88022bf88f4fa78a029138b990cacb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 33121 } }
I20260620 23:33:42.455803 18731 leader_election.cc:302] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae88022bf88f4fa78a029138b990cacb; no voters:
I20260620 23:33:42.456669 18731 leader_election.cc:288] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:42.456871 18736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:42.457850 18736 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [term 1 LEADER]: Becoming Leader. State: Replica: ae88022bf88f4fa78a029138b990cacb, State: Running, Role: LEADER
I20260620 23:33:42.458600 18736 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88022bf88f4fa78a029138b990cacb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 33121 } }
I20260620 23:33:42.459393 18731 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:42.461818 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae88022bf88f4fa78a029138b990cacb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88022bf88f4fa78a029138b990cacb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 33121 } } }
I20260620 23:33:42.462286 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:42.462014 18738 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae88022bf88f4fa78a029138b990cacb. Latest consensus state: current_term: 1 leader_uuid: "ae88022bf88f4fa78a029138b990cacb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae88022bf88f4fa78a029138b990cacb" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 33121 } } }
I20260620 23:33:42.462509 18738 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:42.465072 18742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:42.472626 18742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:42.483585 18742 catalog_manager.cc:1347] Generated new cluster ID: 222d71b5ab284b2fbe028495baf8c580
I20260620 23:33:42.483742 18742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:42.501438 18742 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:42.503176 18742 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:42.513513 18742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb: Generated new TSK 0
I20260620 23:33:42.514660 18742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:33:42.597483 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:33121
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:42.969362 18757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:42.969705 18757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:42.969852 18757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:42.977278 18757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:42.977401 18757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:42.977514 18757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:42.987509 18757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:42.988767 18757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:42.990106 18757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:43.003978 18763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:43.004004 18765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:43.004277 18762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:43.010955 18757 server_base.cc:1034] running on GCE node
I20260620 23:33:43.011993 18757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:43.013449 18757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:43.014750 18757 hybrid_clock.cc:648] HybridClock initialized: now 1781998423014688 us; error 54 us; skew 500 ppm
I20260620 23:33:43.015163 18757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:43 dist-test-slave-v34r krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:43.019261 18757 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:33:43.021371 18757 webserver.cc:460] Webserver started at http://127.0.96.193:34227/ using document root <none> and password file <none>
I20260620 23:33:43.022374 18757 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:43.022497 18757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:43.022830 18757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:43.026618 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "323f55f7b10f4345af92f73ff534931c"
format_stamp: "Formatted at 2026-06-20 23:33:43 on dist-test-slave-v34r"
server_key: "80a1e1d8e6810ec780850e466e984c0e"
server_key_iv: "d4b3b4313e3373c0b3cd8a9d06718089"
server_key_version: "encryptionkey@0"
I20260620 23:33:43.027590 18757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "323f55f7b10f4345af92f73ff534931c"
format_stamp: "Formatted at 2026-06-20 23:33:43 on dist-test-slave-v34r"
server_key: "80a1e1d8e6810ec780850e466e984c0e"
server_key_iv: "d4b3b4313e3373c0b3cd8a9d06718089"
server_key_version: "encryptionkey@0"
I20260620 23:33:43.034623 18757 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260620 23:33:43.039546 18772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:43.041610 18757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:33:43.041905 18757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "323f55f7b10f4345af92f73ff534931c"
format_stamp: "Formatted at 2026-06-20 23:33:43 on dist-test-slave-v34r"
server_key: "80a1e1d8e6810ec780850e466e984c0e"
server_key_iv: "d4b3b4313e3373c0b3cd8a9d06718089"
server_key_version: "encryptionkey@0"
I20260620 23:33:43.042214 18757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:43.059333 18757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:43.065073 18757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:43.065421 18757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:43.066810 18757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:43.068434 18757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:43.068521 18757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:43.068624 18757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:43.068687 18757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:43.106956 18757 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:40075
I20260620 23:33:43.107066 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:40075 every 8 connection(s)
I20260620 23:33:43.108453 18757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:43.113083 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18757
I20260620 23:33:43.113301 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:33:43.113926 387 external_mini_cluster.cc:1428] Setting key aa8bcbf2ccab24edaaaf246c44b26624
Jun 20 23:33:43 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:43.135715 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:43.111845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:50425 (local address 127.0.96.254:33121)
0620 23:33:43.112433 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:43.112438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:43.113636 (+ 1198us) server_negotiation.cc:407] Connection header received
0620 23:33:43.116138 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:43.116144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:43.116259 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:43.116402 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:43.118444 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.119275 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.121151 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.121584 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.121738 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:43.126991 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:43.127003 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:43.127012 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:43.127073 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:43.131292 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:43.132379 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:43.132388 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:43.132395 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:43.132605 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:43.133177 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:43.133185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:43.133191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:43.133653 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:43.133829 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:43.134777 (+ 948us) server_negotiation.cc:299] Negotiation successful
0620 23:33:43.135167 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":328,"threads_started":1}
I20260620 23:33:43.136165 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:43.112278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:33121 (local address 127.0.96.193:50425)
0620 23:33:43.113420 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:43.113482 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:43.115785 (+ 2303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:43.116709 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:43.116719 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:43.117480 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:43.118219 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:43.118239 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.119456 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.119467 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:43.120906 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:43.120919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.121855 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.121865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:43.122149 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:43.123099 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:43.123163 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:43.126594 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:43.131576 (+ 4982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:43.131587 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:43.131602 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:43.132181 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:43.132743 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:43.132746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:43.132749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:43.132949 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:43.133822 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:43.133830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:43.134227 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 23:33:43.134485 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 23:33:43.134942 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":424,"threads_started":1}
I20260620 23:33:43.138130 18885 heartbeater.cc:347] Connected to a master server at 127.0.96.254:33121
I20260620 23:33:43.138447 18885 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:43.139219 18885 heartbeater.cc:511] Master 127.0.96.254:33121 requested a full tablet report, sending...
I20260620 23:33:43.141666 18696 ts_manager.cc:194] Registered new tserver with Master: 323f55f7b10f4345af92f73ff534931c (127.0.96.193:40075)
I20260620 23:33:43.143306 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:50425
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:33:43.181187 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:33121
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:43.591566 18891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:43.592023 18891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:43.592206 18891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:43.600346 18891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:43.600565 18891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:43.600781 18891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:43.612499 18891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:43.614348 18891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:43.616024 18891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:43.632540 18896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:43.632572 18899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:43.632658 18897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:43.638989 18891 server_base.cc:1034] running on GCE node
I20260620 23:33:43.639961 18891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:43.641506 18891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:43.642807 18891 hybrid_clock.cc:648] HybridClock initialized: now 1781998423642734 us; error 57 us; skew 500 ppm
I20260620 23:33:43.643225 18891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:43 dist-test-slave-v34r krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:43.647535 18891 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:33:43.649909 18891 webserver.cc:460] Webserver started at http://127.0.96.194:44793/ using document root <none> and password file <none>
I20260620 23:33:43.650935 18891 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:43.651124 18891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:43.651543 18891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:43.655493 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "3f117cc95230451eb849a6712a981707"
format_stamp: "Formatted at 2026-06-20 23:33:43 on dist-test-slave-v34r"
server_key: "ee831a4bbecc79af357c59fd73f1e1d6"
server_key_iv: "792b2d7cae287dc69d3b738eb7554166"
server_key_version: "encryptionkey@0"
I20260620 23:33:43.656639 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "3f117cc95230451eb849a6712a981707"
format_stamp: "Formatted at 2026-06-20 23:33:43 on dist-test-slave-v34r"
server_key: "ee831a4bbecc79af357c59fd73f1e1d6"
server_key_iv: "792b2d7cae287dc69d3b738eb7554166"
server_key_version: "encryptionkey@0"
I20260620 23:33:43.664146 18891 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260620 23:33:43.669286 18906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:43.671285 18891 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:33:43.671553 18891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "3f117cc95230451eb849a6712a981707"
format_stamp: "Formatted at 2026-06-20 23:33:43 on dist-test-slave-v34r"
server_key: "ee831a4bbecc79af357c59fd73f1e1d6"
server_key_iv: "792b2d7cae287dc69d3b738eb7554166"
server_key_version: "encryptionkey@0"
I20260620 23:33:43.671847 18891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:43.710745 18891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:43.715960 18891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:43.716346 18891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:43.717643 18891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:43.719406 18891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:43.719544 18891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:43.719726 18891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:43.719805 18891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:43.762013 18891 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:43309
I20260620 23:33:43.762125 19018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:43309 every 8 connection(s)
I20260620 23:33:43.763697 18891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:43.772403 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 18891
I20260620 23:33:43.772598 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:33:43.773154 387 external_mini_cluster.cc:1428] Setting key c4a9306194e653851f5673d759dbcbfc
Jun 20 23:33:43 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:43.789500 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:43.767266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:38673 (local address 127.0.96.254:33121)
0620 23:33:43.767792 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:43.767798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:43.769205 (+ 1407us) server_negotiation.cc:407] Connection header received
0620 23:33:43.771178 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:43.771184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:43.771282 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:43.771405 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:43.773548 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.774576 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.776686 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.777598 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.777781 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:43.781719 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:43.781729 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:43.781737 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:43.781795 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:43.785829 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:43.786645 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:43.786655 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:43.786661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:43.786873 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:43.787443 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:43.787447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:43.787449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:43.787771 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:43.787910 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:43.788856 (+ 946us) server_negotiation.cc:299] Negotiation successful
0620 23:33:43.789169 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
I20260620 23:33:43.790776 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:43.768078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:33121 (local address 127.0.96.194:38673)
0620 23:33:43.768941 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:43.769053 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:43.770893 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:43.771662 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:43.771670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:43.772397 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:43.773267 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:43.773294 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.774767 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.774772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:43.776491 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:43.776503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:43.777863 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:43.777868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:43.778069 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:43.778779 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:43.778807 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:43.781430 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:43.786014 (+ 4584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:43.786021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:43.786035 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:43.786481 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:43.787007 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:43.787010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:43.787012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:43.787299 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:43.787920 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:43.787930 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:43.788393 (+ 463us) client_negotiation.cc:769] Sending connection context
0620 23:33:43.788726 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 23:33:43.789344 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":347,"threads_started":1}
I20260620 23:33:43.792549 19019 heartbeater.cc:347] Connected to a master server at 127.0.96.254:33121
I20260620 23:33:43.793067 19019 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:43.794023 19019 heartbeater.cc:511] Master 127.0.96.254:33121 requested a full tablet report, sending...
I20260620 23:33:43.796015 18696 ts_manager.cc:194] Registered new tserver with Master: 3f117cc95230451eb849a6712a981707 (127.0.96.194:43309)
I20260620 23:33:43.797389 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:38673
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:33:43.839988 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:33121
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:41253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:33:44.146642 18885 heartbeater.cc:503] Master 127.0.96.254:33121 was elected leader, sending a full tablet report...
W20260620 23:33:44.201210 19025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:44.201498 19025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:44.201596 19025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:44.209107 19025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:44.209239 19025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:44.209354 19025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:44.219403 19025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:41253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:33121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:44.220708 19025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:44.222100 19025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:44.237353 19031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:44.237358 19030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:44.237380 19033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:44.238368 19025 server_base.cc:1034] running on GCE node
I20260620 23:33:44.239176 19025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:44.240343 19025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:44.241631 19025 hybrid_clock.cc:648] HybridClock initialized: now 1781998424241569 us; error 64 us; skew 500 ppm
I20260620 23:33:44.242046 19025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:44 dist-test-slave-v34r krb5kdc[18651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:44.246261 19025 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:33:44.248394 19025 webserver.cc:460] Webserver started at http://127.0.96.195:35217/ using document root <none> and password file <none>
I20260620 23:33:44.249413 19025 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:44.249619 19025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:44.249969 19025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:44.253935 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "3b146887a5624a9a849d59951ed75d2c"
format_stamp: "Formatted at 2026-06-20 23:33:44 on dist-test-slave-v34r"
server_key: "d0256db44e2af9ee814f498ef292cfed"
server_key_iv: "bd8b8e10cf6ad2e8d2ba5f4035a5464a"
server_key_version: "encryptionkey@0"
I20260620 23:33:44.254989 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "3b146887a5624a9a849d59951ed75d2c"
format_stamp: "Formatted at 2026-06-20 23:33:44 on dist-test-slave-v34r"
server_key: "d0256db44e2af9ee814f498ef292cfed"
server_key_iv: "bd8b8e10cf6ad2e8d2ba5f4035a5464a"
server_key_version: "encryptionkey@0"
I20260620 23:33:44.262909 19025 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20260620 23:33:44.268697 19040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:44.270991 19025 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260620 23:33:44.271307 19025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "3b146887a5624a9a849d59951ed75d2c"
format_stamp: "Formatted at 2026-06-20 23:33:44 on dist-test-slave-v34r"
server_key: "d0256db44e2af9ee814f498ef292cfed"
server_key_iv: "bd8b8e10cf6ad2e8d2ba5f4035a5464a"
server_key_version: "encryptionkey@0"
I20260620 23:33:44.271625 19025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:44.289479 19025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:44.295583 19025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:44.296008 19025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:44.297515 19025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:44.299350 19025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:44.299476 19025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:44.299646 19025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:44.299737 19025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:44.341696 19025 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:39941
I20260620 23:33:44.341720 19152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:39941 every 8 connection(s)
I20260620 23:33:44.343355 19025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:44.352181 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19025
I20260620 23:33:44.352458 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:33:44.353096 387 external_mini_cluster.cc:1428] Setting key fa0f479e6400d3c4ab6563a4d8b8e5c7
Jun 20 23:33:44 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:44.368974 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.346509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:55229 (local address 127.0.96.254:33121)
0620 23:33:44.347051 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.347056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.348482 (+ 1426us) server_negotiation.cc:407] Connection header received
0620 23:33:44.350706 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.350716 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.350808 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.350930 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:44.352603 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.353545 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.356239 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.356940 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.357202 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.361829 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:44.361834 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:44.361837 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:44.361875 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:44.364764 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.365987 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.365991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.365994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.366150 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.366847 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.366851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.366853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.367174 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:44.367377 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.368277 (+ 900us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.368523 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
I20260620 23:33:44.370095 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.347264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:33121 (local address 127.0.96.195:55229)
0620 23:33:44.348272 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.348329 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:44.350374 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:44.351187 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:44.351195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:44.351766 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:44.352380 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.352429 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.353796 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.353802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.355968 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.355991 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.357319 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.357327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.357592 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.358481 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:44.358508 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:44.361558 (+ 3050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:44.365060 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:44.365076 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:44.365096 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:44.365776 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:44.366320 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:44.366324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:44.366328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:44.366672 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:44.367379 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:44.367393 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:44.367873 (+ 480us) client_negotiation.cc:769] Sending connection context
0620 23:33:44.368177 (+ 304us) client_negotiation.cc:240] Negotiation successful
0620 23:33:44.368783 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":416,"threads_started":1}
I20260620 23:33:44.371560 19153 heartbeater.cc:347] Connected to a master server at 127.0.96.254:33121
I20260620 23:33:44.371861 19153 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:44.372619 19153 heartbeater.cc:511] Master 127.0.96.254:33121 requested a full tablet report, sending...
I20260620 23:33:44.374765 18696 ts_manager.cc:194] Registered new tserver with Master: 3b146887a5624a9a849d59951ed75d2c (127.0.96.195:39941)
I20260620 23:33:44.376179 18696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:55229
I20260620 23:33:44.383025 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:33:44.403316 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.388334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.0.96.254:33121)
0620 23:33:44.388647 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.388657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.389105 (+ 448us) server_negotiation.cc:407] Connection header received
0620 23:33:44.389333 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.389338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.389407 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.389517 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:44.391135 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.391879 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.393388 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.393833 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.393976 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.396959 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:44.396965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:44.396968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:44.397067 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:44.400149 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.401182 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.401188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.401192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.401321 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.401798 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.401802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.401804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.402138 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:44.402417 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.402815 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.403005 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 23:33:44.409265 18696 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:33:44.419291 18696 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:33:44.459625 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.434952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:43309 (local address 127.0.0.1:52388)
0620 23:33:44.449123 (+ 14171us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.449148 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:44.449309 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:44.450637 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:44.450649 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:44.450698 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:44.451176 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.451193 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.453265 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.453272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.456079 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.456094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.458402 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.458408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.458622 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.458695 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 23:33:44.458835 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 23:33:44.459097 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14043,"mutex_wait_us":6,"thread_start_us":13851,"threads_started":1}
I20260620 23:33:44.459646 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.435231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.96.194:43309)
0620 23:33:44.447264 (+ 12033us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.447277 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.449461 (+ 2184us) server_negotiation.cc:407] Connection header received
0620 23:33:44.449618 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.449631 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.449993 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.450254 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:44.451394 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.452991 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.456343 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.458158 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.458321 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.458604 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.458929 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.459162 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11826,"thread_start_us":11665,"threads_started":1}
I20260620 23:33:44.460141 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.449284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40380 (local address 127.0.96.195:39941)
0620 23:33:44.451582 (+ 2298us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.451591 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.452206 (+ 615us) server_negotiation.cc:407] Connection header received
0620 23:33:44.452327 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.452335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.452548 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.452804 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:44.453466 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.454724 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.457355 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.458580 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.458705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.458951 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.459549 (+ 598us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.459759 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2082,"thread_start_us":1894,"threads_started":1}
I20260620 23:33:44.461233 19171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.450237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:39941 (local address 127.0.0.1:40380)
0620 23:33:44.451903 (+ 1666us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.451928 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:44.452059 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:44.452809 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:44.452814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:44.452839 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:44.453262 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.453273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.454916 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.454920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.457147 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.457159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.459100 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.459105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.459293 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.459327 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 23:33:44.459406 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 23:33:44.460754 (+ 1348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"mutex_wait_us":4,"thread_start_us":1385,"threads_started":1}
I20260620 23:33:44.461539 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.434710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43810)
0620 23:33:44.448002 (+ 13292us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.448067 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:44.448381 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:44.452094 (+ 3713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:44.452099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:44.452127 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:44.452457 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.452468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.455611 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.455618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.458074 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.458092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.460581 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.460587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.460791 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.460823 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 23:33:44.460923 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 23:33:44.461157 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12772,"mutex_wait_us":2,"thread_start_us":12607,"threads_started":1}
I20260620 23:33:44.461576 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.434952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.0.96.193:40075)
0620 23:33:44.451241 (+ 16289us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.451253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.451298 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:33:44.451423 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.451433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.451785 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.452065 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:44.452684 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.455403 (+ 2719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.458346 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.460306 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.460465 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.460711 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.461033 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.461222 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15998,"thread_start_us":15727,"threads_started":1}
I20260620 23:33:44.464843 18954 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc02316ac634547b0cd42179d9002b8 (DEFAULT_TABLE table=test-table [id=8c0aabb3c92f46beaee7114346acbe1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:44.465871 19088 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc02316ac634547b0cd42179d9002b8 (DEFAULT_TABLE table=test-table [id=8c0aabb3c92f46beaee7114346acbe1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:44.465871 18820 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc02316ac634547b0cd42179d9002b8 (DEFAULT_TABLE table=test-table [id=8c0aabb3c92f46beaee7114346acbe1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:44.466619 18954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc02316ac634547b0cd42179d9002b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:44.467387 18820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc02316ac634547b0cd42179d9002b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:44.467410 19088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc02316ac634547b0cd42179d9002b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:44.482530 19172 tablet_bootstrap.cc:492] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Bootstrap starting.
I20260620 23:33:44.486958 19172 tablet_bootstrap.cc:654] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:44.488299 19172 log.cc:824] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:44.488804 19174 tablet_bootstrap.cc:492] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Bootstrap starting.
I20260620 23:33:44.488804 19173 tablet_bootstrap.cc:492] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Bootstrap starting.
I20260620 23:33:44.492121 19172 tablet_bootstrap.cc:492] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: No bootstrap required, opened a new log
I20260620 23:33:44.492540 19172 ts_tablet_manager.cc:1397] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260620 23:33:44.493996 19174 tablet_bootstrap.cc:654] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:44.493995 19173 tablet_bootstrap.cc:654] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:44.495307 19173 log.cc:824] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:44.495306 19174 log.cc:824] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:44.498965 19173 tablet_bootstrap.cc:492] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: No bootstrap required, opened a new log
I20260620 23:33:44.499313 19173 ts_tablet_manager.cc:1397] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:33:44.499729 19174 tablet_bootstrap.cc:492] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: No bootstrap required, opened a new log
I20260620 23:33:44.499991 19174 ts_tablet_manager.cc:1397] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:33:44.502192 19172 raft_consensus.cc:348] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.502723 19172 raft_consensus.cc:374] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:44.502851 19172 raft_consensus.cc:729] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f117cc95230451eb849a6712a981707, State: Initialized, Role: FOLLOWER
I20260620 23:33:44.503643 19172 consensus_queue.cc:260] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.506155 19172 ts_tablet_manager.cc:1428] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 23:33:44.506822 19019 heartbeater.cc:503] Master 127.0.96.254:33121 was elected leader, sending a full tablet report...
I20260620 23:33:44.510426 19173 raft_consensus.cc:348] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.510426 19174 raft_consensus.cc:348] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.510931 19174 raft_consensus.cc:374] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:44.511041 19174 raft_consensus.cc:729] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b146887a5624a9a849d59951ed75d2c, State: Initialized, Role: FOLLOWER
I20260620 23:33:44.511119 19173 raft_consensus.cc:374] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:44.511237 19173 raft_consensus.cc:729] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 323f55f7b10f4345af92f73ff534931c, State: Initialized, Role: FOLLOWER
I20260620 23:33:44.511969 19174 consensus_queue.cc:260] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.511969 19173 consensus_queue.cc:260] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.514050 19153 heartbeater.cc:503] Master 127.0.96.254:33121 was elected leader, sending a full tablet report...
I20260620 23:33:44.514225 19173 ts_tablet_manager.cc:1428] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 23:33:44.514418 19174 ts_tablet_manager.cc:1428] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260620 23:33:44.516405 19020 tablet.cc:2369] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:44.547240 19179 raft_consensus.cc:482] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:33:44.547575 19179 raft_consensus.cc:504] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.550065 19179 leader_election.cc:288] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b146887a5624a9a849d59951ed75d2c (127.0.96.195:39941), 3f117cc95230451eb849a6712a981707 (127.0.96.194:43309)
I20260620 23:33:44.560432 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.550749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:39249 (local address 127.0.96.195:39941)
0620 23:33:44.550968 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.550975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.552612 (+ 1637us) server_negotiation.cc:407] Connection header received
0620 23:33:44.552820 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.552824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.552892 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.552973 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:44.553582 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.554624 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.557436 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.558915 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.559138 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.559232 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.559945 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.560103 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 23:33:44.560447 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.550736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:47363 (local address 127.0.96.194:43309)
0620 23:33:44.550968 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.550976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.551912 (+ 936us) server_negotiation.cc:407] Connection header received
0620 23:33:44.552102 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.552111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.552231 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.552380 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:44.553148 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.554263 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.557436 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.558893 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.559138 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.559232 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.559917 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.560152 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:33:44.560536 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.550830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:39941 (local address 127.0.96.193:39249)
0620 23:33:44.552485 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.552506 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:44.552675 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:44.553046 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:44.553053 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:44.553087 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:44.553402 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.553411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.554773 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.554778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.557176 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.557189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.559253 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.559260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.559564 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.559694 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 23:33:44.559886 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:33:44.559987 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"mutex_wait_us":169,"thread_start_us":510,"threads_started":1}
I20260620 23:33:44.560575 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.550831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:43309 (local address 127.0.96.193:47363)
0620 23:33:44.551744 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.551765 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:44.551924 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:44.552434 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:44.552439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:44.552476 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:44.552966 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.552982 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.554441 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.554446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.557112 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:44.557143 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.559253 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.559260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:44.559565 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.559704 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 23:33:44.559816 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 23:33:44.559975 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":177,"spinlock_wait_cycles":31488,"thread_start_us":1283,"threads_started":1}
I20260620 23:33:44.562077 18974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc02316ac634547b0cd42179d9002b8" candidate_uuid: "323f55f7b10f4345af92f73ff534931c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f117cc95230451eb849a6712a981707" is_pre_election: true
I20260620 23:33:44.562077 19108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc02316ac634547b0cd42179d9002b8" candidate_uuid: "323f55f7b10f4345af92f73ff534931c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b146887a5624a9a849d59951ed75d2c" is_pre_election: true
I20260620 23:33:44.562809 19108 raft_consensus.cc:2454] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323f55f7b10f4345af92f73ff534931c in term 0.
I20260620 23:33:44.562809 18974 raft_consensus.cc:2454] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 323f55f7b10f4345af92f73ff534931c in term 0.
I20260620 23:33:44.563781 18776 leader_election.cc:302] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 323f55f7b10f4345af92f73ff534931c, 3b146887a5624a9a849d59951ed75d2c, 3f117cc95230451eb849a6712a981707; no voters:
I20260620 23:33:44.564389 19179 raft_consensus.cc:2781] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:33:44.564579 19179 raft_consensus.cc:482] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:33:44.564754 19179 raft_consensus.cc:3037] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:44.567289 19179 raft_consensus.cc:504] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.568009 19179 leader_election.cc:288] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [CANDIDATE]: Term 1 election: Requested vote from peers 3b146887a5624a9a849d59951ed75d2c (127.0.96.195:39941), 3f117cc95230451eb849a6712a981707 (127.0.96.194:43309)
I20260620 23:33:44.568701 19108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc02316ac634547b0cd42179d9002b8" candidate_uuid: "323f55f7b10f4345af92f73ff534931c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b146887a5624a9a849d59951ed75d2c"
I20260620 23:33:44.568722 18974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc02316ac634547b0cd42179d9002b8" candidate_uuid: "323f55f7b10f4345af92f73ff534931c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f117cc95230451eb849a6712a981707"
I20260620 23:33:44.569028 18974 raft_consensus.cc:3037] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:44.569029 19108 raft_consensus.cc:3037] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:44.572769 18974 raft_consensus.cc:2454] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323f55f7b10f4345af92f73ff534931c in term 1.
I20260620 23:33:44.572768 19108 raft_consensus.cc:2454] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 323f55f7b10f4345af92f73ff534931c in term 1.
I20260620 23:33:44.573504 18776 leader_election.cc:302] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 323f55f7b10f4345af92f73ff534931c, 3b146887a5624a9a849d59951ed75d2c; no voters:
I20260620 23:33:44.573917 19179 raft_consensus.cc:2781] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:44.574440 19179 raft_consensus.cc:686] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 1 LEADER]: Becoming Leader. State: Replica: 323f55f7b10f4345af92f73ff534931c, State: Running, Role: LEADER
I20260620 23:33:44.575048 19179 consensus_queue.cc:237] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } }
I20260620 23:33:44.580610 18696 catalog_manager.cc:5495] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c reported cstate change: term changed from 0 to 1, leader changed from <none> to 323f55f7b10f4345af92f73ff534931c (127.0.96.193). New cstate: current_term: 1 leader_uuid: "323f55f7b10f4345af92f73ff534931c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "323f55f7b10f4345af92f73ff534931c" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 40075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 } health_report { overall_health: UNKNOWN } } }
W20260620 23:33:44.595463 19154 tablet.cc:2369] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:44.611640 18886 tablet.cc:2369] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:44.615177 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.608022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.0.96.193:40075)
0620 23:33:44.608480 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.608486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.608511 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:33:44.608635 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.608639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.608759 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.608944 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:33:44.609526 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.610347 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.611966 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.612715 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.612871 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.613553 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:33:44.614212 (+ 659us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:33:44.614399 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.614742 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.614815 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:33:44.626137 18974 raft_consensus.cc:1261] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 1 FOLLOWER]: Refusing update from remote peer 323f55f7b10f4345af92f73ff534931c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:44.626137 19108 raft_consensus.cc:1261] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 1 FOLLOWER]: Refusing update from remote peer 323f55f7b10f4345af92f73ff534931c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:44.627765 19183 consensus_queue.cc:1035] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f117cc95230451eb849a6712a981707" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:44.628196 19179 consensus_queue.cc:1035] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b146887a5624a9a849d59951ed75d2c" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 39941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:44.643067 19186 mvcc.cc:205] Tried to move back new op lower bound from 7299065547252903936 to 7299065547063681024. Current Snapshot: MvccSnapshot[applied={T|T < 7299065547252903936}]
I20260620 23:33:44.644809 19188 mvcc.cc:205] Tried to move back new op lower bound from 7299065547252903936 to 7299065547063681024. Current Snapshot: MvccSnapshot[applied={T|T < 7299065547252903936}]
I20260620 23:33:44.661718 18696 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:44.661978 18696 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:44.666793 18696 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae88022bf88f4fa78a029138b990cacb: Sending DeleteTablet for 3 replicas of tablet 5dc02316ac634547b0cd42179d9002b8
I20260620 23:33:44.668272 18954 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc02316ac634547b0cd42179d9002b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:44 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:52388
I20260620 23:33:44.668483 18820 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc02316ac634547b0cd42179d9002b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:44 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:43810
I20260620 23:33:44.669173 19088 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc02316ac634547b0cd42179d9002b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:44 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:40380
I20260620 23:33:44.669564 19196 tablet_replica.cc:330] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: stopping tablet replica
I20260620 23:33:44.670185 19196 raft_consensus.cc:2229] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:44.670845 19196 raft_consensus.cc:2258] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:44.672495 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18757
I20260620 23:33:44.673799 19196 ts_tablet_manager.cc:1905] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:44.677371 19197 tablet_replica.cc:330] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: stopping tablet replica
I20260620 23:33:44.677591 19198 tablet_replica.cc:330] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: stopping tablet replica
I20260620 23:33:44.678155 19198 raft_consensus.cc:2229] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:44.678824 19198 raft_consensus.cc:2258] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:44.681489 19197 raft_consensus.cc:2229] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:33:44.681733 19198 ts_tablet_manager.cc:1905] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:44.682507 19197 raft_consensus.cc:2258] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:44.686178 19196 ts_tablet_manager.cc:1918] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:44.688448 19197 ts_tablet_manager.cc:1905] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:44.690901 19198 ts_tablet_manager.cc:1918] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:44.691358 19198 log.cc:1196] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/5dc02316ac634547b0cd42179d9002b8
I20260620 23:33:44.692193 19198 ts_tablet_manager.cc:1939] T 5dc02316ac634547b0cd42179d9002b8 P 3b146887a5624a9a849d59951ed75d2c: Deleting consensus metadata
Jun 20 23:33:44 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:33:44.695012 18683 catalog_manager.cc:4841] TS 3b146887a5624a9a849d59951ed75d2c (127.0.96.195:39941): tablet 5dc02316ac634547b0cd42179d9002b8 (table test-table [id=8c0aabb3c92f46beaee7114346acbe1e]) successfully deleted
I20260620 23:33:44.695549 19196 log.cc:1196] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/5dc02316ac634547b0cd42179d9002b8
I20260620 23:33:44.696275 19196 ts_tablet_manager.cc:1939] T 5dc02316ac634547b0cd42179d9002b8 P 3f117cc95230451eb849a6712a981707: Deleting consensus metadata
I20260620 23:33:44.698508 18682 catalog_manager.cc:4841] TS 3f117cc95230451eb849a6712a981707 (127.0.96.194:43309): tablet 5dc02316ac634547b0cd42179d9002b8 (table test-table [id=8c0aabb3c92f46beaee7114346acbe1e]) successfully deleted
I20260620 23:33:44.703192 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.675006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.0.96.193:40075)
0620 23:33:44.675273 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.675282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.681541 (+ 6259us) server_negotiation.cc:407] Connection header received
0620 23:33:44.681849 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.681855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.681952 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.682038 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:44.683926 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.685123 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.687427 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.688177 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.688299 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.694666 (+ 6367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:44.694685 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:44.694708 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:44.694770 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:44.699314 (+ 4544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.700384 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.700395 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.700397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.700626 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.701154 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.701160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.701170 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.701661 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:44.702082 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.702401 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.702726 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:33:44.703405 19197 ts_tablet_manager.cc:1918] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:44.703701 19197 log.cc:1196] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/5dc02316ac634547b0cd42179d9002b8
I20260620 23:33:44.704118 18880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:44.819463 19197 ts_tablet_manager.cc:1939] T 5dc02316ac634547b0cd42179d9002b8 P 323f55f7b10f4345af92f73ff534931c: Deleting consensus metadata
I20260620 23:33:44.820362 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18757
I20260620 23:33:44.842991 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18891
W20260620 23:33:44.842983 18683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:40075 (error 108)
I20260620 23:33:44.845239 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.844691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43832)
0620 23:33:44.844892 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.845073 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:33:44.845518 18683 catalog_manager.cc:4584] TS 323f55f7b10f4345af92f73ff534931c (127.0.96.193:40075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dc02316ac634547b0cd42179d9002b8: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Jun 20 23:33:44 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:33:44.859802 19169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.843935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.0.96.194:43309)
0620 23:33:44.844175 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.844182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.844338 (+ 156us) server_negotiation.cc:407] Connection header received
0620 23:33:44.844551 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.844555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.844647 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.844772 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:44.846316 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.847057 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.848303 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.848690 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.848821 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.853394 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:44.853410 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:44.853427 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:44.853465 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:44.856644 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.857532 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.857540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.857542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.857670 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.858049 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.858054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.858056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.858469 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:44.858808 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.859125 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.859366 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:33:44.860656 19014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:44.899809 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.899160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43836)
0620 23:33:44.899466 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.899593 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 23:33:44.901165 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.900659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43842)
0620 23:33:44.900848 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.900939 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 23:33:44.957267 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18891
I20260620 23:33:44.962585 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.961714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43848)
0620 23:33:44.962138 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.962316 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 23:33:44.963977 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.963453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43860)
0620 23:33:44.963671 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:44.963788 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 23:33:44.979687 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19025
Jun 20 23:33:44 dist-test-slave-v34r krb5kdc[18651](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:33:44.999579 19170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:44.980754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.0.96.195:39941)
0620 23:33:44.981049 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:44.981057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:44.981090 (+ 33us) server_negotiation.cc:407] Connection header received
0620 23:33:44.981292 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:44.981297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:44.981390 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:44.981497 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:44.983178 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.984012 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.985639 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:44.986122 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:44.986267 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:44.992240 (+ 5973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:44.992259 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:44.992276 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:44.992318 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:44.995770 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.996771 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.996779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.996782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.996933 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:44.997748 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:44.997752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:44.997754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:44.998221 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:44.998466 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:44.998875 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 23:33:44.999100 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:33:45.000379 19148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:45.044960 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:45.044321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43864)
0620 23:33:45.044620 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:45.044734 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260620 23:33:45.046487 19166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:45.046009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:40075 (local address 127.0.0.1:43870)
0620 23:33:45.046251 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:45.046331 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 23:33:45.095701 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19025
I20260620 23:33:45.116107 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 18664
I20260620 23:33:45.117785 18726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:45.234144 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 18664
2026-06-20T23:33:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4729 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T23:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:45Z Disabled control of system clock
I20260620 23:33:45.329720 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36979
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:33:45.680626 19208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:45.680938 19208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:45.681120 19208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:45.688661 19208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:45.688789 19208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:45.688829 19208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:45.688930 19208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:45.689059 19208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:45.698753 19208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:45.700176 19208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:45.702023 19208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:45.715735 19214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:45.715694 19216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:45.715806 19213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:45.717051 19208 server_base.cc:1034] running on GCE node
I20260620 23:33:45.718003 19208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:45.719897 19208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:45.721252 19208 hybrid_clock.cc:648] HybridClock initialized: now 1781998425721181 us; error 60 us; skew 500 ppm
I20260620 23:33:45.721724 19208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:45.724707 19208 webserver.cc:460] Webserver started at http://127.0.96.254:33153/ using document root <none> and password file <none>
I20260620 23:33:45.725569 19208 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:45.725679 19208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:45.726049 19208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:45.729223 19208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "efc448d0ffeb45b59b55f33d8c92c9af"
format_stamp: "Formatted at 2026-06-20 23:33:45 on dist-test-slave-v34r"
server_key: "5d2c319a8cd693ba944b93741da53ebc"
server_key_iv: "ec0623fea4be3567f1fd23ae83d891e2"
server_key_version: "encryptionkey@0"
I20260620 23:33:45.730110 19208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "efc448d0ffeb45b59b55f33d8c92c9af"
format_stamp: "Formatted at 2026-06-20 23:33:45 on dist-test-slave-v34r"
server_key: "5d2c319a8cd693ba944b93741da53ebc"
server_key_iv: "ec0623fea4be3567f1fd23ae83d891e2"
server_key_version: "encryptionkey@0"
I20260620 23:33:45.735960 19208 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 23:33:45.740620 19222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:45.742491 19208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:45.742736 19208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "efc448d0ffeb45b59b55f33d8c92c9af"
format_stamp: "Formatted at 2026-06-20 23:33:45 on dist-test-slave-v34r"
server_key: "5d2c319a8cd693ba944b93741da53ebc"
server_key_iv: "ec0623fea4be3567f1fd23ae83d891e2"
server_key_version: "encryptionkey@0"
I20260620 23:33:45.742987 19208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:45.764469 19208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:45.765626 19208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:45.765957 19208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:45.789212 19208 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36979
I20260620 23:33:45.789244 19273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36979 every 8 connection(s)
I20260620 23:33:45.790776 19208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:45.791404 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19208
I20260620 23:33:45.791615 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:45.792366 387 external_mini_cluster.cc:1428] Setting key 77061bb0a6fcb990be61b95e378f1496
I20260620 23:33:45.796120 19274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:45.811538 19277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:45.794759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.96.254:36979)
0620 23:33:45.796228 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:45.796246 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:45.796304 (+ 58us) server_negotiation.cc:407] Connection header received
0620 23:33:45.798940 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:45.798968 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:45.799642 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:45.800167 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:45.800948 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:45.803240 (+ 2292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:45.805293 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:45.806415 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:45.806631 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:45.808222 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:45.808235 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:45.808255 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:45.809152 (+ 897us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:45.809271 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:45.809305 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:45.809470 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:45.809776 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 23:33:45.810179 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":463,"threads_started":1}
I20260620 23:33:45.812574 19274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af: Bootstrap starting.
I20260620 23:33:45.816821 19274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:45.817981 19274 log.cc:824] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:45.821246 19274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af: No bootstrap required, opened a new log
I20260620 23:33:45.828958 19274 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36979 } }
I20260620 23:33:45.829444 19274 raft_consensus.cc:374] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:45.829551 19274 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc448d0ffeb45b59b55f33d8c92c9af, State: Initialized, Role: FOLLOWER
I20260620 23:33:45.830133 19274 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36979 } }
I20260620 23:33:45.830381 19274 raft_consensus.cc:388] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:45.830528 19274 raft_consensus.cc:482] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:45.830705 19274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:45.832945 19274 raft_consensus.cc:504] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36979 } }
I20260620 23:33:45.833566 19274 leader_election.cc:302] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efc448d0ffeb45b59b55f33d8c92c9af; no voters:
I20260620 23:33:45.834348 19274 leader_election.cc:288] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:45.834692 19279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:45.835741 19279 raft_consensus.cc:686] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [term 1 LEADER]: Becoming Leader. State: Replica: efc448d0ffeb45b59b55f33d8c92c9af, State: Running, Role: LEADER
I20260620 23:33:45.836654 19279 consensus_queue.cc:237] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36979 } }
I20260620 23:33:45.836898 19274 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:45.840780 19280 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36979 } } }
I20260620 23:33:45.841580 19280 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:45.841149 19281 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [sys.catalog]: SysCatalogTable state changed. Reason: New leader efc448d0ffeb45b59b55f33d8c92c9af. Latest consensus state: current_term: 1 leader_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efc448d0ffeb45b59b55f33d8c92c9af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36979 } } }
I20260620 23:33:45.842056 19281 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:45.844192 19289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:45.850373 19289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:45.861596 19289 catalog_manager.cc:1347] Generated new cluster ID: 97644a3a670742afb1c8212902ff2f03
I20260620 23:33:45.861779 19289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:45.874432 19289 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:45.876467 19289 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:45.893462 19289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af: Generated new TSK 0
I20260620 23:33:45.894713 19289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:33:45.906607 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36979
--builtin_ntp_servers=127.0.96.212:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:33:46.334013 19298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:46.334420 19298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:46.334589 19298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:46.343756 19298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:46.343895 19298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:46.344103 19298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:46.355582 19298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:46.357443 19298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:46.359899 19298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:46.377254 19303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:46.377270 19304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:46.377437 19306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:46.378224 19298 server_base.cc:1034] running on GCE node
I20260620 23:33:46.379091 19298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:46.380304 19298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:46.381601 19298 hybrid_clock.cc:648] HybridClock initialized: now 1781998426381483 us; error 72 us; skew 500 ppm
I20260620 23:33:46.382169 19298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:46.385879 19298 webserver.cc:460] Webserver started at http://127.0.96.193:43317/ using document root <none> and password file <none>
I20260620 23:33:46.386848 19298 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:46.387001 19298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:46.387436 19298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:46.391266 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48"
format_stamp: "Formatted at 2026-06-20 23:33:46 on dist-test-slave-v34r"
server_key: "2c6febb7a4b15030802733399af36efd"
server_key_iv: "ad99286471012a1ee3548d70ddd3d435"
server_key_version: "encryptionkey@0"
I20260620 23:33:46.392395 19298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48"
format_stamp: "Formatted at 2026-06-20 23:33:46 on dist-test-slave-v34r"
server_key: "2c6febb7a4b15030802733399af36efd"
server_key_iv: "ad99286471012a1ee3548d70ddd3d435"
server_key_version: "encryptionkey@0"
I20260620 23:33:46.399997 19298 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260620 23:33:46.405155 19312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:46.407121 19298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 23:33:46.407377 19298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48"
format_stamp: "Formatted at 2026-06-20 23:33:46 on dist-test-slave-v34r"
server_key: "2c6febb7a4b15030802733399af36efd"
server_key_iv: "ad99286471012a1ee3548d70ddd3d435"
server_key_version: "encryptionkey@0"
I20260620 23:33:46.407694 19298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:46.434315 19298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:46.435485 19298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:46.435883 19298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:46.437292 19298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:46.439288 19298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:46.439400 19298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:46.439584 19298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:46.439697 19298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:46.481155 19298 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:36203
I20260620 23:33:46.481271 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:36203 every 8 connection(s)
I20260620 23:33:46.482865 19298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:46.490573 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19298
I20260620 23:33:46.490749 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:33:46.491351 387 external_mini_cluster.cc:1428] Setting key 0645c19d8e9b7a1aaa0d1913b0d944d7
I20260620 23:33:46.495230 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36979
--builtin_ntp_servers=127.0.96.212:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:33:46.501204 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:46.486434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:58949 (local address 127.0.96.254:36979)
0620 23:33:46.487058 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:46.487064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:46.488505 (+ 1441us) server_negotiation.cc:407] Connection header received
0620 23:33:46.491068 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:46.491075 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:46.491201 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:46.491350 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:46.492955 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.494053 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.497091 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.497602 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.497736 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:46.500085 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:46.500091 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:46.500094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:46.500119 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:46.500212 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:46.500228 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:46.500326 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:46.500703 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 23:33:46.500804 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":304,"threads_started":1}
I20260620 23:33:46.503252 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:46.487009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36979 (local address 127.0.96.193:58949)
0620 23:33:46.488277 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:46.488329 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:46.490495 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:46.491685 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:46.491697 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:46.491833 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:46.492714 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:46.492747 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.494266 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.494273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:46.496801 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:46.496827 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.497933 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.497944 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:46.498383 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:46.499467 (+ 1084us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:46.499473 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:46.499562 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:46.499749 (+ 187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:46.500368 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:46.500382 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:46.500396 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 23:33:46.500657 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 23:33:46.501319 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":349,"threads_started":1}
I20260620 23:33:46.505335 19425 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36979
I20260620 23:33:46.505746 19425 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:46.506631 19425 heartbeater.cc:511] Master 127.0.96.254:36979 requested a full tablet report, sending...
I20260620 23:33:46.509778 19239 ts_manager.cc:194] Registered new tserver with Master: f4284bdb3cb64a3f96a2c7bc0c073f48 (127.0.96.193:36203)
I20260620 23:33:46.511646 19239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.193:58949
W20260620 23:33:46.849743 19429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:46.850024 19429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:46.850158 19429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:46.857438 19429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:46.857548 19429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:46.857708 19429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:46.866921 19429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:46.868251 19429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:46.870059 19429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:46.884608 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:46.884752 19434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:46.884790 19435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:46.886044 19429 server_base.cc:1034] running on GCE node
I20260620 23:33:46.886824 19429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:46.887953 19429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:46.889225 19429 hybrid_clock.cc:648] HybridClock initialized: now 1781998426889182 us; error 81 us; skew 500 ppm
I20260620 23:33:46.889587 19429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:46.892311 19429 webserver.cc:460] Webserver started at http://127.0.96.194:34439/ using document root <none> and password file <none>
I20260620 23:33:46.893185 19429 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:46.893322 19429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:46.893724 19429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:46.896763 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "91921fd05f274574bbc313d03307f8a1"
format_stamp: "Formatted at 2026-06-20 23:33:46 on dist-test-slave-v34r"
server_key: "e3b81f27da6fae68a28f991f42664ba7"
server_key_iv: "58ce478f6da713bed993e65fede0e3b5"
server_key_version: "encryptionkey@0"
I20260620 23:33:46.897707 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "91921fd05f274574bbc313d03307f8a1"
format_stamp: "Formatted at 2026-06-20 23:33:46 on dist-test-slave-v34r"
server_key: "e3b81f27da6fae68a28f991f42664ba7"
server_key_iv: "58ce478f6da713bed993e65fede0e3b5"
server_key_version: "encryptionkey@0"
I20260620 23:33:46.903642 19429 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260620 23:33:46.907855 19443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:46.909654 19429 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260620 23:33:46.909849 19429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "91921fd05f274574bbc313d03307f8a1"
format_stamp: "Formatted at 2026-06-20 23:33:46 on dist-test-slave-v34r"
server_key: "e3b81f27da6fae68a28f991f42664ba7"
server_key_iv: "58ce478f6da713bed993e65fede0e3b5"
server_key_version: "encryptionkey@0"
I20260620 23:33:46.910132 19429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:46.929488 19429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:46.930374 19429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:46.930696 19429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:46.931769 19429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:46.933393 19429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:46.933517 19429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:46.933653 19429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:46.933748 19429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:46.972918 19429 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:39681
I20260620 23:33:46.973106 19555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:39681 every 8 connection(s)
I20260620 23:33:46.974363 19429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:46.981215 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19429
I20260620 23:33:46.981439 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:33:46.981954 387 external_mini_cluster.cc:1428] Setting key c992350df045844288a5b335684c618d
I20260620 23:33:46.985612 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36979
--builtin_ntp_servers=127.0.96.212:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:33:46.990072 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:46.977765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:39555 (local address 127.0.96.254:36979)
0620 23:33:46.977961 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:46.977965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:46.979689 (+ 1724us) server_negotiation.cc:407] Connection header received
0620 23:33:46.981710 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:46.981716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:46.981809 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:46.981923 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:46.983153 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.984384 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.986689 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.987150 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.987310 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:46.989103 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:46.989109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:46.989112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:46.989137 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:46.989178 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:46.989192 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:46.989302 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:46.989558 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 23:33:46.989693 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:33:46.991465 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:46.978389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36979 (local address 127.0.96.194:39555)
0620 23:33:46.979495 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:46.979544 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:46.981378 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:46.982181 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:46.982190 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:46.982286 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:46.982877 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:46.982901 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.984677 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.984682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:46.986482 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:46.986501 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:46.987433 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:46.987441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:46.987729 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:46.988629 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:46.988632 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:46.988686 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:46.988802 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:46.989300 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:46.989312 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:46.989323 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 23:33:46.989548 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 23:33:46.990099 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":336,"threads_started":1}
I20260620 23:33:46.993538 19556 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36979
I20260620 23:33:46.994038 19556 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:46.995065 19556 heartbeater.cc:511] Master 127.0.96.254:36979 requested a full tablet report, sending...
I20260620 23:33:46.997115 19239 ts_manager.cc:194] Registered new tserver with Master: 91921fd05f274574bbc313d03307f8a1 (127.0.96.194:39681)
I20260620 23:33:46.998416 19239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.194:39555
W20260620 23:33:47.326402 19559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:47.326697 19559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:47.326785 19559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:47.334450 19559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:47.334573 19559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:47.334683 19559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:47.344494 19559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:47.345855 19559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:47.347852 19559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:47.363231 19564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:47.363332 19567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:47.363343 19565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:47.364548 19559 server_base.cc:1034] running on GCE node
I20260620 23:33:47.365374 19559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:47.366540 19559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:47.367794 19559 hybrid_clock.cc:648] HybridClock initialized: now 1781998427367713 us; error 75 us; skew 500 ppm
I20260620 23:33:47.368232 19559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:47.375792 19559 webserver.cc:460] Webserver started at http://127.0.96.195:42873/ using document root <none> and password file <none>
I20260620 23:33:47.376606 19559 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:47.376745 19559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:47.377207 19559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:47.380246 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "9df27e77f29a4341afe2e81d47dae816"
format_stamp: "Formatted at 2026-06-20 23:33:47 on dist-test-slave-v34r"
server_key: "49af3589472f3ab95b5d6b4ee2164282"
server_key_iv: "990cba1d405a7445a715cef53fa4a8c8"
server_key_version: "encryptionkey@0"
I20260620 23:33:47.381165 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "9df27e77f29a4341afe2e81d47dae816"
format_stamp: "Formatted at 2026-06-20 23:33:47 on dist-test-slave-v34r"
server_key: "49af3589472f3ab95b5d6b4ee2164282"
server_key_iv: "990cba1d405a7445a715cef53fa4a8c8"
server_key_version: "encryptionkey@0"
I20260620 23:33:47.387612 19559 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 23:33:47.393450 19573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:47.395318 19559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:33:47.395534 19559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "9df27e77f29a4341afe2e81d47dae816"
format_stamp: "Formatted at 2026-06-20 23:33:47 on dist-test-slave-v34r"
server_key: "49af3589472f3ab95b5d6b4ee2164282"
server_key_iv: "990cba1d405a7445a715cef53fa4a8c8"
server_key_version: "encryptionkey@0"
I20260620 23:33:47.395885 19559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:47.411164 19559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:47.412431 19559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:47.412844 19559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:47.414250 19559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:47.416186 19559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:47.416289 19559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:47.416477 19559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:47.416569 19559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:47.456951 19559 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:36771
I20260620 23:33:47.457170 19685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:36771 every 8 connection(s)
I20260620 23:33:47.458758 19559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:47.463996 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19559
I20260620 23:33:47.464228 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:33:47.464887 387 external_mini_cluster.cc:1428] Setting key 63851fa36d05109371774164c83c68a8
I20260620 23:33:47.473711 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.462148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:37979 (local address 127.0.96.254:36979)
0620 23:33:47.462386 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.462391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.464402 (+ 2011us) server_negotiation.cc:407] Connection header received
0620 23:33:47.466724 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.466729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.466822 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.466921 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:47.468379 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.469304 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.470869 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.471273 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.471415 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.472817 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:47.472822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:47.472825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:47.472845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:47.472879 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:47.472892 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:47.472992 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.473249 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.473355 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 23:33:47.474696 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.462810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36979 (local address 127.0.96.195:37979)
0620 23:33:47.464134 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.464207 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:47.466387 (+ 2180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:47.467184 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:47.467194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:47.467298 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:47.468165 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.468191 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.469456 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.469461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.470715 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.470726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.471504 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.471512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.471746 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.472469 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:47.472471 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:47.472518 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:47.472597 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:47.473020 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:47.473032 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:47.473048 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 23:33:47.473239 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 23:33:47.473626 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":353,"threads_started":1}
I20260620 23:33:47.476265 19686 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36979
I20260620 23:33:47.476537 19686 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:47.477298 19686 heartbeater.cc:511] Master 127.0.96.254:36979 requested a full tablet report, sending...
I20260620 23:33:47.478814 19239 ts_manager.cc:194] Registered new tserver with Master: 9df27e77f29a4341afe2e81d47dae816 (127.0.96.195:36771)
I20260620 23:33:47.479858 19239 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.195:37979
I20260620 23:33:47.482254 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:33:47.495579 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.487937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.0.96.254:36979)
0620 23:33:47.488250 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.488261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.488725 (+ 464us) server_negotiation.cc:407] Connection header received
0620 23:33:47.488953 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.488958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.489100 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.489227 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:47.489768 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.490553 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.492062 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.492534 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.492700 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.494192 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:47.494204 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:47.494213 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:47.494247 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:47.494318 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:47.494344 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:47.494625 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.494979 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.495087 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 23:33:47.500408 19239 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:33:47.507530 19239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:33:47.516403 19425 heartbeater.cc:503] Master 127.0.96.254:36979 was elected leader, sending a full tablet report...
I20260620 23:33:47.549283 19700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.523090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:39681 (local address 127.0.0.1:42706)
0620 23:33:47.535194 (+ 12104us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.535264 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:47.535572 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:47.539860 (+ 4288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:47.539879 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:47.539928 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:47.540516 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.540540 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.542764 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.542771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.545579 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.545598 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.547789 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.547796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.548225 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.548276 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:33:47.548459 (+ 183us) client_negotiation.cc:240] Negotiation successful
0620 23:33:47.548740 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11709,"mutex_wait_us":168,"thread_start_us":21021,"threads_started":1}
I20260620 23:33:47.549393 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.523276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.0.96.194:39681)
0620 23:33:47.538052 (+ 14776us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.538069 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.538125 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:33:47.538289 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.538303 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.538792 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.539130 (+ 338us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:47.540818 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.542575 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.545874 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.547526 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.547685 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.548017 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.548592 (+ 575us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.548856 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14459,"thread_start_us":16159,"threads_started":1}
I20260620 23:33:47.551699 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.524109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.0.96.193:36203)
0620 23:33:47.540211 (+ 16102us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.540221 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.541849 (+ 1628us) server_negotiation.cc:407] Connection header received
0620 23:33:47.541995 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.542060 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.542415 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.542667 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:47.544106 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.545776 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.548781 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.550191 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.550324 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.550619 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.551176 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.551331 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15837,"thread_start_us":16099,"threads_started":1}
I20260620 23:33:47.551710 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.523256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41678)
0620 23:33:47.541511 (+ 18255us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.541539 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:47.541699 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:47.543209 (+ 1510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:47.543215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:47.543256 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:47.543801 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.543814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.545962 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.545971 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.548495 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.548510 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.550772 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.550780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.550987 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.551016 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 23:33:47.551134 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 23:33:47.551284 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18077,"mutex_wait_us":2,"thread_start_us":11277,"threads_started":1}
I20260620 23:33:47.554781 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.523667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:36771 (local address 127.0.0.1:60286)
0620 23:33:47.544470 (+ 20803us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.544494 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:47.544641 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:47.545715 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:47.545722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:47.545758 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:47.546306 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.546322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.548412 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.548420 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.551794 (+ 3374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.551810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.553898 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.553903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.554047 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.554074 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:33:47.554285 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 23:33:47.554451 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20634,"thread_start_us":17476,"threads_started":1}
I20260620 23:33:47.556035 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.524425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.0.96.195:36771)
0620 23:33:47.543112 (+ 18687us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.543123 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.544649 (+ 1526us) server_negotiation.cc:407] Connection header received
0620 23:33:47.544842 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.544852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.545193 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.545421 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:47.546568 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.548248 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.552107 (+ 3859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.553643 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.553802 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.554111 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.554369 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.555489 (+ 1120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16423,"thread_start_us":16252,"threads_started":1}
I20260620 23:33:47.554894 19491 tablet_service.cc:1467] Processing CreateTablet for tablet d5999e7cdb9b4bd988dbc758c9b5643c (DEFAULT_TABLE table=test-table [id=953334c51b3942ae97f824a1b37d09b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:47.557981 19491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5999e7cdb9b4bd988dbc758c9b5643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:47.558521 19360 tablet_service.cc:1467] Processing CreateTablet for tablet d5999e7cdb9b4bd988dbc758c9b5643c (DEFAULT_TABLE table=test-table [id=953334c51b3942ae97f824a1b37d09b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:47.560312 19360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5999e7cdb9b4bd988dbc758c9b5643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:47.560489 19621 tablet_service.cc:1467] Processing CreateTablet for tablet d5999e7cdb9b4bd988dbc758c9b5643c (DEFAULT_TABLE table=test-table [id=953334c51b3942ae97f824a1b37d09b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:47.562091 19621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5999e7cdb9b4bd988dbc758c9b5643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:47.572741 19704 tablet_bootstrap.cc:492] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Bootstrap starting.
I20260620 23:33:47.578042 19706 tablet_bootstrap.cc:492] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Bootstrap starting.
I20260620 23:33:47.578101 19704 tablet_bootstrap.cc:654] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:47.580118 19704 log.cc:824] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:47.582181 19705 tablet_bootstrap.cc:492] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Bootstrap starting.
I20260620 23:33:47.584182 19706 tablet_bootstrap.cc:654] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:47.584832 19704 tablet_bootstrap.cc:492] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: No bootstrap required, opened a new log
I20260620 23:33:47.585239 19704 ts_tablet_manager.cc:1397] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260620 23:33:47.586057 19706 log.cc:824] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:47.587406 19705 tablet_bootstrap.cc:654] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:47.588958 19705 log.cc:824] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:47.590189 19706 tablet_bootstrap.cc:492] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: No bootstrap required, opened a new log
I20260620 23:33:47.590565 19706 ts_tablet_manager.cc:1397] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260620 23:33:47.592689 19705 tablet_bootstrap.cc:492] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: No bootstrap required, opened a new log
I20260620 23:33:47.593065 19705 ts_tablet_manager.cc:1397] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260620 23:33:47.593827 19704 raft_consensus.cc:348] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.594220 19704 raft_consensus.cc:374] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:47.594416 19704 raft_consensus.cc:729] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91921fd05f274574bbc313d03307f8a1, State: Initialized, Role: FOLLOWER
I20260620 23:33:47.595059 19704 consensus_queue.cc:260] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.598341 19556 heartbeater.cc:503] Master 127.0.96.254:36979 was elected leader, sending a full tablet report...
I20260620 23:33:47.598791 19704 ts_tablet_manager.cc:1428] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260620 23:33:47.602237 19706 raft_consensus.cc:348] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.602926 19706 raft_consensus.cc:374] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:47.603065 19706 raft_consensus.cc:729] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9df27e77f29a4341afe2e81d47dae816, State: Initialized, Role: FOLLOWER
I20260620 23:33:47.603884 19706 consensus_queue.cc:260] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.605317 19686 heartbeater.cc:503] Master 127.0.96.254:36979 was elected leader, sending a full tablet report...
I20260620 23:33:47.607005 19705 raft_consensus.cc:348] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.607587 19705 raft_consensus.cc:374] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:47.607707 19705 raft_consensus.cc:729] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4284bdb3cb64a3f96a2c7bc0c073f48, State: Initialized, Role: FOLLOWER
I20260620 23:33:47.608008 19706 ts_tablet_manager.cc:1428] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260620 23:33:47.608567 19705 consensus_queue.cc:260] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.610889 19705 ts_tablet_manager.cc:1428] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Time spent starting tablet: real 0.018s user 0.011s sys 0.003s
I20260620 23:33:47.660034 19712 raft_consensus.cc:482] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:33:47.660506 19712 raft_consensus.cc:504] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.662773 19712 leader_election.cc:288] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91921fd05f274574bbc313d03307f8a1 (127.0.96.194:39681), 9df27e77f29a4341afe2e81d47dae816 (127.0.96.195:36771)
I20260620 23:33:47.671301 19713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.663611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:36771 (local address 127.0.96.193:52377)
0620 23:33:47.664662 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.664684 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:47.664893 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:47.665410 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:47.665415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:47.665447 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:47.665987 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.666009 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.667190 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.667198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.668837 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.668851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.670376 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.670383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.670589 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.670617 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 23:33:47.670739 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 23:33:47.670877 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":34,"thread_start_us":595,"threads_started":1}
I20260620 23:33:47.671324 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.663623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:52377 (local address 127.0.96.195:36771)
0620 23:33:47.663880 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.663887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.664792 (+ 905us) server_negotiation.cc:407] Connection header received
0620 23:33:47.665156 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.665166 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.665270 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.665354 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:47.666194 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.667017 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.669094 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.670122 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.670286 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.670388 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.670830 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.671014 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:33:47.671629 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.663586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:42807 (local address 127.0.96.194:39681)
0620 23:33:47.663858 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.663865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.665037 (+ 1172us) server_negotiation.cc:407] Connection header received
0620 23:33:47.665268 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.665273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.665400 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.665512 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:47.666376 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.667131 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.669430 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.670263 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.670443 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.670521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.671176 (+ 655us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.671359 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 23:33:47.672273 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.663924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:39681 (local address 127.0.96.193:42807)
0620 23:33:47.664821 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.664838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:47.665100 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:47.665548 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:47.665552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:47.665590 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:47.666061 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.666070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.667346 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.667351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.669207 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:47.669225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.670514 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.670519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:47.670923 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.670962 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 23:33:47.671745 (+ 783us) client_negotiation.cc:240] Negotiation successful
0620 23:33:47.671918 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":145,"thread_start_us":488,"threads_started":1}
I20260620 23:33:47.673156 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5999e7cdb9b4bd988dbc758c9b5643c" candidate_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df27e77f29a4341afe2e81d47dae816" is_pre_election: true
I20260620 23:33:47.673604 19511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5999e7cdb9b4bd988dbc758c9b5643c" candidate_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91921fd05f274574bbc313d03307f8a1" is_pre_election: true
I20260620 23:33:47.673740 19641 raft_consensus.cc:2454] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4284bdb3cb64a3f96a2c7bc0c073f48 in term 0.
I20260620 23:33:47.674175 19511 raft_consensus.cc:2454] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4284bdb3cb64a3f96a2c7bc0c073f48 in term 0.
I20260620 23:33:47.674527 19314 leader_election.cc:302] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9df27e77f29a4341afe2e81d47dae816, f4284bdb3cb64a3f96a2c7bc0c073f48; no voters:
I20260620 23:33:47.675021 19712 raft_consensus.cc:2781] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:33:47.675433 19712 raft_consensus.cc:482] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:33:47.675616 19712 raft_consensus.cc:3037] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:47.678416 19712 raft_consensus.cc:504] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.679353 19712 leader_election.cc:288] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [CANDIDATE]: Term 1 election: Requested vote from peers 91921fd05f274574bbc313d03307f8a1 (127.0.96.194:39681), 9df27e77f29a4341afe2e81d47dae816 (127.0.96.195:36771)
I20260620 23:33:47.679965 19511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5999e7cdb9b4bd988dbc758c9b5643c" candidate_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91921fd05f274574bbc313d03307f8a1"
I20260620 23:33:47.680269 19511 raft_consensus.cc:3037] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:47.680230 19641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5999e7cdb9b4bd988dbc758c9b5643c" candidate_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9df27e77f29a4341afe2e81d47dae816"
I20260620 23:33:47.680490 19641 raft_consensus.cc:3037] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:47.684005 19641 raft_consensus.cc:2454] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4284bdb3cb64a3f96a2c7bc0c073f48 in term 1.
I20260620 23:33:47.683985 19511 raft_consensus.cc:2454] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4284bdb3cb64a3f96a2c7bc0c073f48 in term 1.
I20260620 23:33:47.684643 19314 leader_election.cc:302] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9df27e77f29a4341afe2e81d47dae816, f4284bdb3cb64a3f96a2c7bc0c073f48; no voters:
I20260620 23:33:47.685065 19712 raft_consensus.cc:2781] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:47.685672 19712 raft_consensus.cc:686] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 1 LEADER]: Becoming Leader. State: Replica: f4284bdb3cb64a3f96a2c7bc0c073f48, State: Running, Role: LEADER
I20260620 23:33:47.686276 19712 consensus_queue.cc:237] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } }
I20260620 23:33:47.691604 19239 catalog_manager.cc:5495] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4284bdb3cb64a3f96a2c7bc0c073f48 (127.0.96.193). New cstate: current_term: 1 leader_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4284bdb3cb64a3f96a2c7bc0c073f48" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 36203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 } health_report { overall_health: UNKNOWN } } }
W20260620 23:33:47.711113 19687 tablet.cc:2369] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:47.719547 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.713553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41688 (local address 127.0.96.193:36203)
0620 23:33:47.713756 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.713761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.713782 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:33:47.714000 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.714007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.714140 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.714338 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:33:47.714836 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.715592 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.717146 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.717572 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.717694 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.718316 (+ 622us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:33:47.718850 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:33:47.718990 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.719295 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.719340 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 23:33:47.726734 19557 tablet.cc:2369] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:47.730894 19641 raft_consensus.cc:1261] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 1 FOLLOWER]: Refusing update from remote peer f4284bdb3cb64a3f96a2c7bc0c073f48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:47.730894 19511 raft_consensus.cc:1261] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 1 FOLLOWER]: Refusing update from remote peer f4284bdb3cb64a3f96a2c7bc0c073f48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:47.732388 19712 consensus_queue.cc:1035] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91921fd05f274574bbc313d03307f8a1" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 39681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:47.732836 19715 consensus_queue.cc:1035] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9df27e77f29a4341afe2e81d47dae816" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 23:33:47.736722 19426 tablet.cc:2369] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:47.751046 19719 mvcc.cc:205] Tried to move back new op lower bound from 7299065559971033088 to 7299065559807238144. Current Snapshot: MvccSnapshot[applied={T|T < 7299065559971033088}]
I20260620 23:33:47.751736 19720 mvcc.cc:205] Tried to move back new op lower bound from 7299065559971033088 to 7299065559807238144. Current Snapshot: MvccSnapshot[applied={T|T < 7299065559971033088}]
I20260620 23:33:47.768289 19238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:47.768676 19238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:47.773921 19238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efc448d0ffeb45b59b55f33d8c92c9af: Sending DeleteTablet for 3 replicas of tablet d5999e7cdb9b4bd988dbc758c9b5643c
I20260620 23:33:47.775596 19491 tablet_service.cc:1514] Processing DeleteTablet for tablet d5999e7cdb9b4bd988dbc758c9b5643c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:47 UTC) from {username='slave'} at 127.0.0.1:42706
I20260620 23:33:47.775622 19360 tablet_service.cc:1514] Processing DeleteTablet for tablet d5999e7cdb9b4bd988dbc758c9b5643c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:47 UTC) from {username='slave'} at 127.0.0.1:41678
I20260620 23:33:47.775878 19621 tablet_service.cc:1514] Processing DeleteTablet for tablet d5999e7cdb9b4bd988dbc758c9b5643c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:47 UTC) from {username='slave'} at 127.0.0.1:60286
I20260620 23:33:47.778079 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19298
I20260620 23:33:47.786928 19727 tablet_replica.cc:330] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: stopping tablet replica
I20260620 23:33:47.787190 19729 tablet_replica.cc:330] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: stopping tablet replica
I20260620 23:33:47.788065 19729 raft_consensus.cc:2229] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:47.788399 19727 raft_consensus.cc:2229] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:47.789713 19727 raft_consensus.cc:2258] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:47.790607 19728 tablet_replica.cc:330] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: stopping tablet replica
I20260620 23:33:47.791642 19728 raft_consensus.cc:2229] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:33:47.792834 19728 raft_consensus.cc:2258] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:47.794803 19727 ts_tablet_manager.cc:1905] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:47.795037 19729 raft_consensus.cc:2258] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:47.795910 19702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.779595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.0.96.193:36203)
0620 23:33:47.779788 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.779795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.785252 (+ 5457us) server_negotiation.cc:407] Connection header received
0620 23:33:47.785553 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.785559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.785685 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.785820 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:47.787580 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.788880 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.790968 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.791678 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.791780 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.793471 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:47.793484 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:47.793508 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:47.794383 (+ 875us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:47.794453 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:47.794479 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:47.794565 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.795353 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.795486 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:33:47.797680 19420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:47.800027 19729 ts_tablet_manager.cc:1905] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:47.805495 19727 ts_tablet_manager.cc:1918] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:47.805768 19727 log.cc:1196] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/d5999e7cdb9b4bd988dbc758c9b5643c
I20260620 23:33:47.806344 19727 ts_tablet_manager.cc:1939] T d5999e7cdb9b4bd988dbc758c9b5643c P 91921fd05f274574bbc313d03307f8a1: Deleting consensus metadata
I20260620 23:33:47.808005 19729 ts_tablet_manager.cc:1918] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:47.808326 19729 log.cc:1196] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/d5999e7cdb9b4bd988dbc758c9b5643c
I20260620 23:33:47.808568 19225 catalog_manager.cc:4841] TS 91921fd05f274574bbc313d03307f8a1 (127.0.96.194:39681): tablet d5999e7cdb9b4bd988dbc758c9b5643c (table test-table [id=953334c51b3942ae97f824a1b37d09b7]) successfully deleted
I20260620 23:33:47.809098 19729 ts_tablet_manager.cc:1939] T d5999e7cdb9b4bd988dbc758c9b5643c P 9df27e77f29a4341afe2e81d47dae816: Deleting consensus metadata
I20260620 23:33:47.811426 19224 catalog_manager.cc:4841] TS 9df27e77f29a4341afe2e81d47dae816 (127.0.96.195:36771): tablet d5999e7cdb9b4bd988dbc758c9b5643c (table test-table [id=953334c51b3942ae97f824a1b37d09b7]) successfully deleted
I20260620 23:33:47.798501 19728 ts_tablet_manager.cc:1905] T d5999e7cdb9b4bd988dbc758c9b5643c P f4284bdb3cb64a3f96a2c7bc0c073f48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:47.918154 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19298
I20260620 23:33:47.937199 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19429
W20260620 23:33:47.937217 19226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:36203 (error 108)
I20260620 23:33:47.939535 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.938984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41718)
0620 23:33:47.939221 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.939356 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:33:47.939836 19226 catalog_manager.cc:4584] TS f4284bdb3cb64a3f96a2c7bc0c073f48 (127.0.96.193:36203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5999e7cdb9b4bd988dbc758c9b5643c: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
I20260620 23:33:47.945472 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.938268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.96.194:39681)
0620 23:33:47.938514 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:47.938521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:47.938543 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:33:47.938828 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:47.938832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:47.938928 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:47.939050 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:47.939498 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.940490 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.942179 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:47.942599 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:47.942759 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:47.944031 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:47.944040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:47.944053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:47.944635 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:47.944685 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:47.944710 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:47.944850 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:47.945006 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 23:33:47.945120 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:33:47.946146 19551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:47.994458 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.993709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41734)
0620 23:33:47.993994 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.994157 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 23:33:47.995697 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:47.995297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41740)
0620 23:33:47.995470 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:47.995570 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 23:33:48.047360 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19429
I20260620 23:33:48.056870 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:48.056071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41754)
0620 23:33:48.056423 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:48.056602 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 23:33:48.058355 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:48.057840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41768)
0620 23:33:48.058042 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:48.058156 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 23:33:48.068451 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19559
I20260620 23:33:48.076757 19703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:48.069578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.0.96.195:36771)
0620 23:33:48.069822 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:48.069830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:48.069855 (+ 25us) server_negotiation.cc:407] Connection header received
0620 23:33:48.070102 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:48.070108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:48.070243 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:48.070371 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:48.070834 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:48.071652 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:48.073240 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:48.073774 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:48.073919 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:48.075307 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:48.075316 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:48.075331 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:48.075928 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:48.075981 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:48.076001 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:48.076117 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:48.076271 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 23:33:48.076378 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:33:48.077605 19681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:48.139340 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:48.138622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41772)
0620 23:33:48.138946 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:48.139101 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 23:33:48.140789 19698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:48.140274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:36203 (local address 127.0.0.1:41780)
0620 23:33:48.140541 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:48.140640 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 23:33:48.171258 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19559
I20260620 23:33:48.192060 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19208
I20260620 23:33:48.193506 19269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:48.307308 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19208
2026-06-20T23:33:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:33:48 dist-test-slave-v34r krb5kdc[19738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:33:48 dist-test-slave-v34r krb5kdc[19738](info): set up 2 sockets
Jun 20 23:33:48 dist-test-slave-v34r krb5kdc[19738](info): commencing operation
krb5kdc: starting...
W20260620 23:33:49.445547 387 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:33:49 dist-test-slave-v34r krb5kdc[19738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:33:49.743912 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36877
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:50.146735 19751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:50.147058 19751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:50.147204 19751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:50.154846 19751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:50.154976 19751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:50.155076 19751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:50.155192 19751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:50.155283 19751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:50.165750 19751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:50.167167 19751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:50.168545 19751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:50.182643 19759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:50.182821 19757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:50.182852 19756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:50.183892 19751 server_base.cc:1034] running on GCE node
I20260620 23:33:50.184831 19751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:50.186515 19751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:50.187780 19751 hybrid_clock.cc:648] HybridClock initialized: now 1781998430187727 us; error 51 us; skew 500 ppm
I20260620 23:33:50.188165 19751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:50 dist-test-slave-v34r krb5kdc[19738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:50.191781 19751 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:33:50.193781 19751 webserver.cc:460] Webserver started at http://127.0.96.254:38313/ using document root <none> and password file <none>
I20260620 23:33:50.194569 19751 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:50.194718 19751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:50.195077 19751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:50.198269 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "02b73517bb68463abab93305350378a3"
format_stamp: "Formatted at 2026-06-20 23:33:50 on dist-test-slave-v34r"
server_key: "4d0f5cd03f531765bca76a68b40b0b37"
server_key_iv: "ba9c7dcec256849d32cad91baa102b97"
server_key_version: "encryptionkey@0"
I20260620 23:33:50.199059 19751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "02b73517bb68463abab93305350378a3"
format_stamp: "Formatted at 2026-06-20 23:33:50 on dist-test-slave-v34r"
server_key: "4d0f5cd03f531765bca76a68b40b0b37"
server_key_iv: "ba9c7dcec256849d32cad91baa102b97"
server_key_version: "encryptionkey@0"
I20260620 23:33:50.205199 19751 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260620 23:33:50.209398 19766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:50.211196 19751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:33:50.211452 19751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "02b73517bb68463abab93305350378a3"
format_stamp: "Formatted at 2026-06-20 23:33:50 on dist-test-slave-v34r"
server_key: "4d0f5cd03f531765bca76a68b40b0b37"
server_key_iv: "ba9c7dcec256849d32cad91baa102b97"
server_key_version: "encryptionkey@0"
I20260620 23:33:50.211673 19751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:50.235438 19751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:50.240865 19751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:50.241285 19751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:50.263931 19751 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36877
I20260620 23:33:50.264035 19817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36877 every 8 connection(s)
I20260620 23:33:50.265460 19751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:50.270522 19818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:50.275296 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19751
I20260620 23:33:50.275520 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:50.276005 387 external_mini_cluster.cc:1428] Setting key 672576fa15793d4f968d40429e21211d
I20260620 23:33:50.287209 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3: Bootstrap starting.
I20260620 23:33:50.292272 19818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3: Neither blocks nor log segments found. Creating new log.
Jun 20 23:33:50 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:50.293839 19818 log.cc:824] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:50.298945 19818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3: No bootstrap required, opened a new log
I20260620 23:33:50.305486 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:50.278351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.0.96.254:36877)
0620 23:33:50.279356 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:50.279374 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:50.279443 (+ 69us) server_negotiation.cc:407] Connection header received
0620 23:33:50.281889 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:50.281936 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:50.282706 (+ 770us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:50.283315 (+ 609us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:50.284710 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:50.286913 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:50.288469 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:50.289775 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:50.290065 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:50.294812 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:50.294827 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:50.294860 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:50.294949 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:50.299436 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:50.300334 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:50.300344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:50.300355 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:50.300633 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:50.301126 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:50.301131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:50.301135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:50.302034 (+ 899us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:50.302414 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:50.302958 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 23:33:50.303621 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":406,"threads_started":1}
I20260620 23:33:50.310330 19818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b73517bb68463abab93305350378a3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36877 } }
I20260620 23:33:50.310793 19818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:50.310882 19818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b73517bb68463abab93305350378a3, State: Initialized, Role: FOLLOWER
I20260620 23:33:50.311501 19818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b73517bb68463abab93305350378a3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36877 } }
I20260620 23:33:50.311739 19818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:50.311847 19818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:50.312008 19818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:50.315249 19818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b73517bb68463abab93305350378a3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36877 } }
I20260620 23:33:50.316039 19818 leader_election.cc:302] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b73517bb68463abab93305350378a3; no voters:
I20260620 23:33:50.316843 19818 leader_election.cc:288] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:50.317080 19823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:50.318051 19823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [term 1 LEADER]: Becoming Leader. State: Replica: 02b73517bb68463abab93305350378a3, State: Running, Role: LEADER
I20260620 23:33:50.318769 19823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b73517bb68463abab93305350378a3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36877 } }
I20260620 23:33:50.319924 19818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:50.322099 19825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b73517bb68463abab93305350378a3. Latest consensus state: current_term: 1 leader_uuid: "02b73517bb68463abab93305350378a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b73517bb68463abab93305350378a3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36877 } } }
I20260620 23:33:50.322571 19825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:50.322193 19824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b73517bb68463abab93305350378a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b73517bb68463abab93305350378a3" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36877 } } }
I20260620 23:33:50.322733 19824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:50.326785 19829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:50.334540 19829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:50.347044 19829 catalog_manager.cc:1347] Generated new cluster ID: 039ea36422fc4bdebb40cc102c07d898
I20260620 23:33:50.347219 19829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:50.359382 19829 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:50.361537 19829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:50.373678 19829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3: Generated new TSK 0
I20260620 23:33:50.374861 19829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.96.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
Entry for principal kudu/127.0.96.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab.
I20260620 23:33:50.454006 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36877
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:50.898403 19844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:50.898852 19844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:50.899047 19844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:50.908403 19844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:50.908612 19844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:50.908828 19844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:50.921226 19844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.193.keytab
--principal=kudu/127.0.96.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:50.923023 19844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:50.924752 19844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:50.941625 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:50.941962 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:50.941617 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:50.944639 19844 server_base.cc:1034] running on GCE node
I20260620 23:33:50.945508 19844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:50.946759 19844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:50.948105 19844 hybrid_clock.cc:648] HybridClock initialized: now 1781998430948058 us; error 68 us; skew 500 ppm
I20260620 23:33:50.948479 19844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:50 dist-test-slave-v34r krb5kdc[19738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:50.952145 19844 init.cc:377] Logged in from keytab as kudu/127.0.96.193@KRBTEST.COM (short username kudu)
I20260620 23:33:50.954144 19844 webserver.cc:460] Webserver started at http://127.0.96.193:39959/ using document root <none> and password file <none>
I20260620 23:33:50.954993 19844 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:50.955152 19844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:50.955492 19844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:50.958803 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "5ad04a7822f64a9cb4ce9868dbc68b72"
format_stamp: "Formatted at 2026-06-20 23:33:50 on dist-test-slave-v34r"
server_key: "c4f460acf5757fe037e597cb083ef56a"
server_key_iv: "7a1e5b0e41b79d6f6f5bfd7203620438"
server_key_version: "encryptionkey@0"
I20260620 23:33:50.959709 19844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "5ad04a7822f64a9cb4ce9868dbc68b72"
format_stamp: "Formatted at 2026-06-20 23:33:50 on dist-test-slave-v34r"
server_key: "c4f460acf5757fe037e597cb083ef56a"
server_key_iv: "7a1e5b0e41b79d6f6f5bfd7203620438"
server_key_version: "encryptionkey@0"
I20260620 23:33:50.965665 19844 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260620 23:33:50.970185 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:50.972028 19844 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:33:50.972311 19844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "5ad04a7822f64a9cb4ce9868dbc68b72"
format_stamp: "Formatted at 2026-06-20 23:33:50 on dist-test-slave-v34r"
server_key: "c4f460acf5757fe037e597cb083ef56a"
server_key_iv: "7a1e5b0e41b79d6f6f5bfd7203620438"
server_key_version: "encryptionkey@0"
I20260620 23:33:50.972563 19844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:50.994231 19844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:50.999402 19844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:50.999742 19844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:51.000950 19844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:51.002594 19844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:51.002713 19844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:51.002900 19844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:51.002981 19844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:51.043920 19844 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:33459
I20260620 23:33:51.044023 19971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:33459 every 8 connection(s)
I20260620 23:33:51.045360 19844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:51.049446 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19844
I20260620 23:33:51.049645 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:33:51.050132 387 external_mini_cluster.cc:1428] Setting key eede4a86df5f55ca1dcfbde12214df40
Jun 20 23:33:51 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.193@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:51.070659 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:51.048723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:34493 (local address 127.0.96.254:36877)
0620 23:33:51.049406 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:51.049415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:51.051186 (+ 1771us) server_negotiation.cc:407] Connection header received
0620 23:33:51.053140 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:51.053146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:51.053271 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:51.053412 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:51.055366 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.056194 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.058235 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.058703 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.058862 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:51.063404 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:51.063411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:51.063414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:51.063466 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:51.066574 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:51.067666 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:51.067671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:51.067674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:51.067829 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:51.068374 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:51.068378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:51.068380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:51.068720 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:51.068902 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:51.069883 (+ 981us) server_negotiation.cc:299] Negotiation successful
0620 23:33:51.070204 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":333,"threads_started":1}
I20260620 23:33:51.072072 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:51.049485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36877 (local address 127.0.96.193:34493)
0620 23:33:51.050603 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:51.050660 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:51.052741 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:51.053729 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:51.053741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:51.054406 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:51.055131 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:51.055153 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.056412 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.056417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:51.058035 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:51.058051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.059025 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.059033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:51.059339 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:51.060232 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:51.060269 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:51.063061 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:51.066824 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:51.066839 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:51.066868 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:51.067481 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:51.067972 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:51.067977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:51.067981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:51.068225 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:51.068870 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:51.068880 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:51.069437 (+ 557us) client_negotiation.cc:769] Sending connection context
0620 23:33:51.069777 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 23:33:51.070525 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":440,"threads_started":1}
I20260620 23:33:51.074227 19972 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36877
I20260620 23:33:51.074649 19972 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.96.194@KRBTEST.COM; defaulting to no policy
I20260620 23:33:51.075664 19972 heartbeater.cc:511] Master 127.0.96.254:36877 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.194@KRBTEST.COM" created.
I20260620 23:33:51.078887 19783 ts_manager.cc:194] Registered new tserver with Master: 5ad04a7822f64a9cb4ce9868dbc68b72 (127.0.96.193:33459)
I20260620 23:33:51.080700 19783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.193@KRBTEST.COM'} at 127.0.96.193:34493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
Entry for principal kudu/127.0.96.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab.
I20260620 23:33:51.107951 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36877
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:51.469885 19978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:51.470186 19978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:51.470281 19978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:51.477981 19978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:51.478112 19978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:51.478226 19978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:51.487949 19978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.194.keytab
--principal=kudu/127.0.96.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:51.489256 19978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:51.490559 19978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:51.505473 19986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:51.505844 19983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:51.505594 19984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:51.507038 19978 server_base.cc:1034] running on GCE node
I20260620 23:33:51.507903 19978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:51.509373 19978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:51.510622 19978 hybrid_clock.cc:648] HybridClock initialized: now 1781998431510549 us; error 46 us; skew 500 ppm
I20260620 23:33:51.511049 19978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:51 dist-test-slave-v34r krb5kdc[19738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:51.515085 19978 init.cc:377] Logged in from keytab as kudu/127.0.96.194@KRBTEST.COM (short username kudu)
I20260620 23:33:51.517244 19978 webserver.cc:460] Webserver started at http://127.0.96.194:37017/ using document root <none> and password file <none>
I20260620 23:33:51.518148 19978 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:51.518272 19978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:51.518594 19978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:51.522629 19978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "3b97902430d244678099c0fc7b7e7a9c"
format_stamp: "Formatted at 2026-06-20 23:33:51 on dist-test-slave-v34r"
server_key: "0bdce8a663aff35dd6da84cde9cc240d"
server_key_iv: "697c7b2a62475eac41444ed6a8530cca"
server_key_version: "encryptionkey@0"
I20260620 23:33:51.523680 19978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "3b97902430d244678099c0fc7b7e7a9c"
format_stamp: "Formatted at 2026-06-20 23:33:51 on dist-test-slave-v34r"
server_key: "0bdce8a663aff35dd6da84cde9cc240d"
server_key_iv: "697c7b2a62475eac41444ed6a8530cca"
server_key_version: "encryptionkey@0"
I20260620 23:33:51.531018 19978 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 23:33:51.536108 19993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:51.538187 19978 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:33:51.538463 19978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "3b97902430d244678099c0fc7b7e7a9c"
format_stamp: "Formatted at 2026-06-20 23:33:51 on dist-test-slave-v34r"
server_key: "0bdce8a663aff35dd6da84cde9cc240d"
server_key_iv: "697c7b2a62475eac41444ed6a8530cca"
server_key_version: "encryptionkey@0"
I20260620 23:33:51.538744 19978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:51.568377 19978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:51.574450 19978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:51.574828 19978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:51.576285 19978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:51.578454 19978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:51.578608 19978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:51.578786 19978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:51.578867 19978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:51.621445 19978 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:44247
I20260620 23:33:51.621543 20105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:44247 every 8 connection(s)
I20260620 23:33:51.623329 19978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:51.626968 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 19978
I20260620 23:33:51.627189 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:33:51.627826 387 external_mini_cluster.cc:1428] Setting key 21f6c28c4985d977fcf0aee7c3e60e27
Jun 20 23:33:51 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.194@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:51.648751 20108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:51.626531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:35621 (local address 127.0.96.254:36877)
0620 23:33:51.627133 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:51.627141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:51.628713 (+ 1572us) server_negotiation.cc:407] Connection header received
0620 23:33:51.631329 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:51.631335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:51.631436 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:51.631564 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:51.633314 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.634234 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.635925 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.636382 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.636531 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:51.640500 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:51.640506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:51.640509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:51.640547 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:51.643851 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:51.645232 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:51.645238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:51.645241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:51.645446 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:51.646136 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:51.646140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:51.646143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:51.646838 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:51.647022 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:51.648122 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0620 23:33:51.648391 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":341,"threads_started":1}
I20260620 23:33:51.650152 20109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:51.627297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36877 (local address 127.0.96.194:35621)
0620 23:33:51.628440 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:51.628497 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:51.631013 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:51.631853 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:51.631863 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:51.632450 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:51.633120 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:51.633151 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.634411 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.634418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:51.635767 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:51.635780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:51.636642 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:51.636649 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:51.636895 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:51.637729 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:51.637753 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:51.640064 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:51.644151 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:51.644164 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:51.644188 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:51.645060 (+ 872us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:51.645611 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:51.645619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:51.645625 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:51.645940 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:51.647022 (+ 1082us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:51.647034 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:51.647558 (+ 524us) client_negotiation.cc:769] Sending connection context
0620 23:33:51.647909 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 23:33:51.648539 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":490,"threads_started":1}
I20260620 23:33:51.652057 20106 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36877
I20260620 23:33:51.652539 20106 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:51.653609 20106 heartbeater.cc:511] Master 127.0.96.254:36877 requested a full tablet report, sending...
I20260620 23:33:51.655742 19783 ts_manager.cc:194] Registered new tserver with Master: 3b97902430d244678099c0fc7b7e7a9c (127.0.96.194:44247)
I20260620 23:33:51.657173 19783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.194@KRBTEST.COM'} at 127.0.96.194:35621
WARNING: no policy specified for kudu/127.0.96.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
Entry for principal kudu/127.0.96.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab.
I20260620 23:33:51.696035 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36877
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:52.037197 20112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:52.037510 20112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:52.037671 20112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:52.045790 20112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:52.045917 20112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:52.046092 20112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:52.056265 20112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/krb5kdc/kudu_127.0.96.195.keytab
--principal=kudu/127.0.96.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:52.057682 20112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:52.058972 20112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:52.073580 20118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:52.073722 20117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:52.073721 20120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:52.074954 20112 server_base.cc:1034] running on GCE node
I20260620 23:33:52.075625 20112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:52.076689 20112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:52.077926 20112 hybrid_clock.cc:648] HybridClock initialized: now 1781998432077873 us; error 48 us; skew 500 ppm
I20260620 23:33:52.078287 20112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:52 dist-test-slave-v34r krb5kdc[19738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:52.081784 20112 init.cc:377] Logged in from keytab as kudu/127.0.96.195@KRBTEST.COM (short username kudu)
I20260620 23:33:52.083429 20112 webserver.cc:460] Webserver started at http://127.0.96.195:37123/ using document root <none> and password file <none>
I20260620 23:33:52.084157 20112 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:52.084312 20112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:52.084622 20112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:52.084841 19972 heartbeater.cc:503] Master 127.0.96.254:36877 was elected leader, sending a full tablet report...
I20260620 23:33:52.088114 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "02dc62ee8b4b4f31a49bf1c919684d79"
format_stamp: "Formatted at 2026-06-20 23:33:52 on dist-test-slave-v34r"
server_key: "041e2508b475491fcbd6adb0bdaa8cbf"
server_key_iv: "c92efd5fda1cf271807c039981f31706"
server_key_version: "encryptionkey@0"
I20260620 23:33:52.088969 20112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "02dc62ee8b4b4f31a49bf1c919684d79"
format_stamp: "Formatted at 2026-06-20 23:33:52 on dist-test-slave-v34r"
server_key: "041e2508b475491fcbd6adb0bdaa8cbf"
server_key_iv: "c92efd5fda1cf271807c039981f31706"
server_key_version: "encryptionkey@0"
I20260620 23:33:52.094640 20112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 23:33:52.098855 20127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:52.100627 20112 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:33:52.100832 20112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "02dc62ee8b4b4f31a49bf1c919684d79"
format_stamp: "Formatted at 2026-06-20 23:33:52 on dist-test-slave-v34r"
server_key: "041e2508b475491fcbd6adb0bdaa8cbf"
server_key_iv: "c92efd5fda1cf271807c039981f31706"
server_key_version: "encryptionkey@0"
I20260620 23:33:52.101148 20112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:52.114037 20112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:52.118597 20112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:52.118901 20112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:52.119987 20112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:52.121531 20112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:52.121636 20112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:52.121768 20112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:52.121856 20112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:52.162062 20112 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:33113
I20260620 23:33:52.162102 20239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:33113 every 8 connection(s)
I20260620 23:33:52.163439 20112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:52.168042 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20112
I20260620 23:33:52.168272 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:33:52.168926 387 external_mini_cluster.cc:1428] Setting key 2e340f229e5f6335e1fc879a9780a695
Jun 20 23:33:52 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.195@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:33:52.188578 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.166733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:47319 (local address 127.0.96.254:36877)
0620 23:33:52.167329 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.167334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.168461 (+ 1127us) server_negotiation.cc:407] Connection header received
0620 23:33:52.171310 (+ 2849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.171315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.171459 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.171601 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:52.173794 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.174611 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.176545 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.176971 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.177156 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.181522 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:52.181526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:52.181530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:52.181566 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:52.184834 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.186119 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.186123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.186125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.186267 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.186779 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.186782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.186784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.187044 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:52.187159 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.188110 (+ 951us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.188288 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":338,"threads_started":1}
I20260620 23:33:52.190115 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.167104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36877 (local address 127.0.96.195:47319)
0620 23:33:52.168154 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:52.168236 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:52.170909 (+ 2673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:52.171883 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:52.171893 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:52.172739 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:52.173594 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.173619 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.174821 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.174827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.176378 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.176397 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.177304 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.177311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.177583 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.178383 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:33:52.178412 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:52.181276 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:52.185125 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:52.185136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:52.185156 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:52.185967 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:52.186384 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:33:52.186389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:33:52.186391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:33:52.186656 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:33:52.187169 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:52.187177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:52.187686 (+ 509us) client_negotiation.cc:769] Sending connection context
0620 23:33:52.187985 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 23:33:52.188566 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":352,"threads_started":1}
I20260620 23:33:52.192217 20240 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36877
I20260620 23:33:52.192656 20240 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:52.193750 20240 heartbeater.cc:511] Master 127.0.96.254:36877 requested a full tablet report, sending...
I20260620 23:33:52.195809 19783 ts_manager.cc:194] Registered new tserver with Master: 02dc62ee8b4b4f31a49bf1c919684d79 (127.0.96.195:33113)
I20260620 23:33:52.197115 19783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.96.195@KRBTEST.COM'} at 127.0.96.195:47319
I20260620 23:33:52.198889 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:33:52.218969 20242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.203692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.0.96.254:36877)
0620 23:33:52.204033 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.204045 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.204538 (+ 493us) server_negotiation.cc:407] Connection header received
0620 23:33:52.204780 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.204785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.204879 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.205075 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:52.206706 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.207592 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.208972 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.209533 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.209668 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.212223 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:52.212229 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:52.212233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:52.212287 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:52.215681 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.216492 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.216497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.216500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.216673 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.217260 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.217265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.217268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.217706 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:52.217946 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.218272 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.218542 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260620 23:33:52.224416 19783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:33:52.231844 19783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:33:52.271235 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.248019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.0.96.194:44247)
0620 23:33:52.260543 (+ 12524us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.260562 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.261515 (+ 953us) server_negotiation.cc:407] Connection header received
0620 23:33:52.261629 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.261636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.261921 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.262884 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:52.263084 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.264760 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.267749 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.269304 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.270446 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.270658 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.270746 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.270927 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12183,"thread_start_us":11977,"threads_started":1}
I20260620 23:33:52.272025 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.247544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:44247 (local address 127.0.0.1:59926)
0620 23:33:52.261011 (+ 13467us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:52.261056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:52.261298 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:52.262151 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:52.262158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:52.262189 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:52.262664 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.262684 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.264909 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.264916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.267529 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.267546 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.269548 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.269555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.269761 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.269815 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 23:33:52.269919 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 23:33:52.270128 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13146,"thread_start_us":12955,"threads_started":1}
I20260620 23:33:52.277493 20041 tablet_service.cc:1467] Processing CreateTablet for tablet e4089bafe51b43a2a989b04e5472bef5 (DEFAULT_TABLE table=test-table [id=7ec06c812a634446a9676ba371b2426d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:52.279315 20041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4089bafe51b43a2a989b04e5472bef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:52.280242 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.247768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:33459 (local address 127.0.0.1:48826)
0620 23:33:52.263466 (+ 15698us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:52.263490 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:52.263711 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:52.265135 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:52.265144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:52.265192 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:52.265953 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.265971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.270240 (+ 4269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.270246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.274299 (+ 4053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.274337 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.278637 (+ 4300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.278651 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.278982 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.279046 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 23:33:52.279202 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 23:33:52.279641 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15540,"thread_start_us":15320,"threads_started":1}
I20260620 23:33:52.281899 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.247990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.0.96.193:33459)
0620 23:33:52.262894 (+ 14904us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.262903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.263874 (+ 971us) server_negotiation.cc:407] Connection header received
0620 23:33:52.264016 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.264027 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.264357 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.264660 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:52.267658 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.270012 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.274726 (+ 4714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.277561 (+ 2835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.277808 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.278189 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.281032 (+ 2843us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.281371 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14671,"thread_start_us":14524,"threads_started":1}
I20260620 23:33:52.287431 19907 tablet_service.cc:1467] Processing CreateTablet for tablet e4089bafe51b43a2a989b04e5472bef5 (DEFAULT_TABLE table=test-table [id=7ec06c812a634446a9676ba371b2426d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:52.288928 19907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4089bafe51b43a2a989b04e5472bef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:52.291858 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.270163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:33113 (local address 127.0.0.1:40482)
0620 23:33:52.272283 (+ 2120us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:52.272303 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:52.272407 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:52.284094 (+ 11687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:52.284099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:52.284135 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:52.284355 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.284367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.286341 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.286347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.288639 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.288665 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.290949 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.290957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.291168 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.291219 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 23:33:52.291309 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 23:33:52.291489 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1996,"thread_start_us":13249,"threads_started":1}
I20260620 23:33:52.293454 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.264134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.0.96.195:33113)
0620 23:33:52.283196 (+ 19062us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.283209 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.283256 (+ 47us) server_negotiation.cc:407] Connection header received
0620 23:33:52.283395 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.283407 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.283817 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.284576 (+ 759us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:52.284922 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.286147 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.288935 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.290673 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.292391 (+ 1718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.292701 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.292793 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.293051 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18824,"thread_start_us":18612,"threads_started":1}
I20260620 23:33:52.297688 20175 tablet_service.cc:1467] Processing CreateTablet for tablet e4089bafe51b43a2a989b04e5472bef5 (DEFAULT_TABLE table=test-table [id=7ec06c812a634446a9676ba371b2426d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:52.299712 20175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4089bafe51b43a2a989b04e5472bef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:52.303936 20259 tablet_bootstrap.cc:492] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Bootstrap starting.
I20260620 23:33:52.305315 20260 tablet_bootstrap.cc:492] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Bootstrap starting.
I20260620 23:33:52.308482 20259 tablet_bootstrap.cc:654] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:52.309653 20259 log.cc:824] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:52.309720 20260 tablet_bootstrap.cc:654] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:52.311220 20260 log.cc:824] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:52.312857 20259 tablet_bootstrap.cc:492] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: No bootstrap required, opened a new log
I20260620 23:33:52.313175 20259 ts_tablet_manager.cc:1397] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 23:33:52.315166 20260 tablet_bootstrap.cc:492] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: No bootstrap required, opened a new log
I20260620 23:33:52.315527 20260 ts_tablet_manager.cc:1397] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260620 23:33:52.321275 20263 tablet_bootstrap.cc:492] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Bootstrap starting.
I20260620 23:33:52.322129 20259 raft_consensus.cc:348] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.322604 20259 raft_consensus.cc:374] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:52.322707 20259 raft_consensus.cc:729] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b97902430d244678099c0fc7b7e7a9c, State: Initialized, Role: FOLLOWER
I20260620 23:33:52.323414 20259 consensus_queue.cc:260] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.325174 20106 heartbeater.cc:503] Master 127.0.96.254:36877 was elected leader, sending a full tablet report...
I20260620 23:33:52.325424 20259 ts_tablet_manager.cc:1428] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260620 23:33:52.325827 20263 tablet_bootstrap.cc:654] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:52.327165 20263 log.cc:824] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:52.328099 20260 raft_consensus.cc:348] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.328732 20260 raft_consensus.cc:374] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:52.328856 20260 raft_consensus.cc:729] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad04a7822f64a9cb4ce9868dbc68b72, State: Initialized, Role: FOLLOWER
I20260620 23:33:52.329717 20260 consensus_queue.cc:260] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.331332 20263 tablet_bootstrap.cc:492] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: No bootstrap required, opened a new log
I20260620 23:33:52.331640 20263 ts_tablet_manager.cc:1397] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 23:33:52.340689 20260 ts_tablet_manager.cc:1428] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Time spent starting tablet: real 0.025s user 0.018s sys 0.008s
I20260620 23:33:52.340613 20263 raft_consensus.cc:348] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.341199 20263 raft_consensus.cc:374] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:52.341326 20263 raft_consensus.cc:729] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02dc62ee8b4b4f31a49bf1c919684d79, State: Initialized, Role: FOLLOWER
I20260620 23:33:52.342027 20263 consensus_queue.cc:260] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.343441 20240 heartbeater.cc:503] Master 127.0.96.254:36877 was elected leader, sending a full tablet report...
I20260620 23:33:52.344202 20263 ts_tablet_manager.cc:1428] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260620 23:33:52.376122 20107 tablet.cc:2369] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:52.415170 20241 tablet.cc:2369] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:52.548228 19973 tablet.cc:2369] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:52.581743 20267 raft_consensus.cc:482] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:33:52.582118 20267 raft_consensus.cc:504] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.583824 20267 leader_election.cc:288] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ad04a7822f64a9cb4ce9868dbc68b72 (127.0.96.193:33459), 3b97902430d244678099c0fc7b7e7a9c (127.0.96.194:44247)
I20260620 23:33:52.592379 20243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.584628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:33459 (local address 127.0.96.195:52811)
0620 23:33:52.584903 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:52.584928 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:52.585148 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:52.585751 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:52.585758 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:52.585800 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:52.586071 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.586089 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.587623 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.587627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.589641 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.589657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.591624 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.591629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.591769 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.591796 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:33:52.591878 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 23:33:52.592031 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 23:33:52.592443 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.584658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:52811 (local address 127.0.96.193:33459)
0620 23:33:52.584951 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.584961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.585053 (+ 92us) server_negotiation.cc:407] Connection header received
0620 23:33:52.585364 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.585373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.585517 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.585692 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:52.586349 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.587442 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.589912 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.591401 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.591673 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.591782 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.591956 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.592126 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 23:33:52.593120 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.584598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:52161 (local address 127.0.96.194:44247)
0620 23:33:52.584836 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.584842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.586318 (+ 1476us) server_negotiation.cc:407] Connection header received
0620 23:33:52.586509 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.586516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.586611 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.586738 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:52.587229 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.588066 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.590338 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.591832 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.592064 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.592143 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.592469 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.592639 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:33:52.593294 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.585144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:44247 (local address 127.0.96.195:52161)
0620 23:33:52.586103 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:52.586127 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:52.586344 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:52.586767 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:52.586771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:52.586791 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:52.587059 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.587066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.588245 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.588250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.590133 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:52.590143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.592089 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.592094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:52.592272 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.592299 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 23:33:52.592407 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 23:33:52.592901 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"thread_start_us":560,"threads_started":1}
I20260620 23:33:52.593958 19927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4089bafe51b43a2a989b04e5472bef5" candidate_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" is_pre_election: true
I20260620 23:33:52.594560 20061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4089bafe51b43a2a989b04e5472bef5" candidate_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b97902430d244678099c0fc7b7e7a9c" is_pre_election: true
I20260620 23:33:52.594745 19927 raft_consensus.cc:2454] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02dc62ee8b4b4f31a49bf1c919684d79 in term 0.
I20260620 23:33:52.595158 20061 raft_consensus.cc:2454] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02dc62ee8b4b4f31a49bf1c919684d79 in term 0.
I20260620 23:33:52.595860 20130 leader_election.cc:302] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02dc62ee8b4b4f31a49bf1c919684d79, 3b97902430d244678099c0fc7b7e7a9c, 5ad04a7822f64a9cb4ce9868dbc68b72; no voters:
I20260620 23:33:52.596437 20267 raft_consensus.cc:2781] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:33:52.596614 20267 raft_consensus.cc:482] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:33:52.596730 20267 raft_consensus.cc:3037] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:52.599128 20267 raft_consensus.cc:504] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.599939 20267 leader_election.cc:288] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [CANDIDATE]: Term 1 election: Requested vote from peers 5ad04a7822f64a9cb4ce9868dbc68b72 (127.0.96.193:33459), 3b97902430d244678099c0fc7b7e7a9c (127.0.96.194:44247)
I20260620 23:33:52.600543 19927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4089bafe51b43a2a989b04e5472bef5" candidate_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72"
I20260620 23:33:52.600711 20061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4089bafe51b43a2a989b04e5472bef5" candidate_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b97902430d244678099c0fc7b7e7a9c"
I20260620 23:33:52.600855 19927 raft_consensus.cc:3037] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:52.601085 20061 raft_consensus.cc:3037] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:52.603286 19927 raft_consensus.cc:2454] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02dc62ee8b4b4f31a49bf1c919684d79 in term 1.
I20260620 23:33:52.603852 20130 leader_election.cc:302] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02dc62ee8b4b4f31a49bf1c919684d79, 5ad04a7822f64a9cb4ce9868dbc68b72; no voters:
I20260620 23:33:52.603986 20061 raft_consensus.cc:2454] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02dc62ee8b4b4f31a49bf1c919684d79 in term 1.
I20260620 23:33:52.604277 20267 raft_consensus.cc:2781] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:52.604803 20267 raft_consensus.cc:686] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 1 LEADER]: Becoming Leader. State: Replica: 02dc62ee8b4b4f31a49bf1c919684d79, State: Running, Role: LEADER
I20260620 23:33:52.605419 20267 consensus_queue.cc:237] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } }
I20260620 23:33:52.609915 19783 catalog_manager.cc:5495] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02dc62ee8b4b4f31a49bf1c919684d79 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02dc62ee8b4b4f31a49bf1c919684d79" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 33113 } health_report { overall_health: HEALTHY } } }
I20260620 23:33:52.650703 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.644892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.0.96.195:33113)
0620 23:33:52.645145 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.645151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.645211 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:33:52.645334 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.645339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.645442 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.645626 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:33:52.645970 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.646754 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.648161 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.648621 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.648737 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.649325 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:33:52.650068 (+ 743us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:33:52.650213 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.650353 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.650453 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 23:33:52.660589 19927 raft_consensus.cc:1261] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 1 FOLLOWER]: Refusing update from remote peer 02dc62ee8b4b4f31a49bf1c919684d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:52.661024 20061 raft_consensus.cc:1261] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 1 FOLLOWER]: Refusing update from remote peer 02dc62ee8b4b4f31a49bf1c919684d79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:52.661880 20269 consensus_queue.cc:1035] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad04a7822f64a9cb4ce9868dbc68b72" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:52.662341 20267 consensus_queue.cc:1035] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b97902430d244678099c0fc7b7e7a9c" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:52.676182 20272 mvcc.cc:205] Tried to move back new op lower bound from 7299065580164997120 to 7299065579955511296. Current Snapshot: MvccSnapshot[applied={T|T < 7299065580164997120}]
I20260620 23:33:52.692909 19783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:52.693353 19783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:52.697965 19783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02b73517bb68463abab93305350378a3: Sending DeleteTablet for 3 replicas of tablet e4089bafe51b43a2a989b04e5472bef5
I20260620 23:33:52.699549 20041 tablet_service.cc:1514] Processing DeleteTablet for tablet e4089bafe51b43a2a989b04e5472bef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:59926
I20260620 23:33:52.699549 19907 tablet_service.cc:1514] Processing DeleteTablet for tablet e4089bafe51b43a2a989b04e5472bef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:48826
I20260620 23:33:52.699864 20175 tablet_service.cc:1514] Processing DeleteTablet for tablet e4089bafe51b43a2a989b04e5472bef5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:52 UTC) from {username='kudu', principal='kudu/127.0.96.254@KRBTEST.COM'} at 127.0.0.1:40482
I20260620 23:33:52.707402 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19844
I20260620 23:33:52.711778 20284 tablet_replica.cc:330] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: stopping tablet replica
I20260620 23:33:52.712303 20282 tablet_replica.cc:330] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: stopping tablet replica
I20260620 23:33:52.712916 20282 raft_consensus.cc:2229] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:52.713521 20282 raft_consensus.cc:2258] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:52.713474 20284 raft_consensus.cc:2229] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:33:52.714468 20283 tablet_replica.cc:330] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: stopping tablet replica
I20260620 23:33:52.714886 20284 raft_consensus.cc:2258] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:52.714907 20283 raft_consensus.cc:2229] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:52.715557 20283 raft_consensus.cc:2258] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:52.718005 20284 ts_tablet_manager.cc:1905] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:52.718791 20282 ts_tablet_manager.cc:1905] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:52.719580 20283 ts_tablet_manager.cc:1905] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:52.726432 20282 ts_tablet_manager.cc:1918] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:52.726631 20282 log.cc:1196] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/e4089bafe51b43a2a989b04e5472bef5
I20260620 23:33:52.727243 20282 ts_tablet_manager.cc:1939] T e4089bafe51b43a2a989b04e5472bef5 P 3b97902430d244678099c0fc7b7e7a9c: Deleting consensus metadata
I20260620 23:33:52.727422 20283 ts_tablet_manager.cc:1918] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:52.727734 20283 log.cc:1196] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-0/wal/wals/e4089bafe51b43a2a989b04e5472bef5
I20260620 23:33:52.728333 20283 ts_tablet_manager.cc:1939] T e4089bafe51b43a2a989b04e5472bef5 P 5ad04a7822f64a9cb4ce9868dbc68b72: Deleting consensus metadata
I20260620 23:33:52.729513 19770 catalog_manager.cc:4841] TS 3b97902430d244678099c0fc7b7e7a9c (127.0.96.194:44247): tablet e4089bafe51b43a2a989b04e5472bef5 (table test-table [id=7ec06c812a634446a9676ba371b2426d]) successfully deleted
I20260620 23:33:52.730252 19769 catalog_manager.cc:4841] TS 5ad04a7822f64a9cb4ce9868dbc68b72 (127.0.96.193:33459): tablet e4089bafe51b43a2a989b04e5472bef5 (table test-table [id=7ec06c812a634446a9676ba371b2426d]) successfully deleted
Jun 20 23:33:52 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.193@KRBTEST.COM
I20260620 23:33:52.732673 20284 ts_tablet_manager.cc:1918] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:52.732870 20284 log.cc:1196] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/e4089bafe51b43a2a989b04e5472bef5
I20260620 23:33:52.733547 20284 ts_tablet_manager.cc:1939] T e4089bafe51b43a2a989b04e5472bef5 P 02dc62ee8b4b4f31a49bf1c919684d79: Deleting consensus metadata
I20260620 23:33:52.735373 19769 catalog_manager.cc:4841] TS 02dc62ee8b4b4f31a49bf1c919684d79 (127.0.96.195:33113): tablet e4089bafe51b43a2a989b04e5472bef5 (table test-table [id=7ec06c812a634446a9676ba371b2426d]) successfully deleted
I20260620 23:33:52.740254 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.708583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.0.96.193:33459)
0620 23:33:52.708834 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.708841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.710758 (+ 1917us) server_negotiation.cc:407] Connection header received
0620 23:33:52.711785 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.711793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.711945 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.712038 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:52.714343 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.715724 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.718340 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.719113 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.719386 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.731997 (+ 12611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:52.732012 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:52.732031 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:52.732086 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:52.736158 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.737491 (+ 1333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.737499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.737502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.737663 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.738297 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.738301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.738303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.738672 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:52.738926 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.739703 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.739915 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 23:33:52.741194 19967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:52.834950 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19844
I20260620 23:33:52.854269 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19978
Jun 20 23:33:52 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.194@KRBTEST.COM
I20260620 23:33:52.871201 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.855255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.0.96.194:44247)
0620 23:33:52.855454 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.855462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.855483 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:33:52.855704 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.855716 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.855785 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.855895 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:52.857288 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.858027 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.859300 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.859717 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.859853 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:52.864873 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:52.864890 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:52.864906 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:52.864951 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:52.868072 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.868841 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.868848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.868851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.869169 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:52.869611 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:52.869617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:52.869621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:52.870005 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:52.870318 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:52.870612 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 23:33:52.870834 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 23:33:52.872131 20101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:52.962540 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19978
I20260620 23:33:52.992199 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20112
Jun 20 23:33:52 dist-test-slave-v34r krb5kdc[19738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.195@KRBTEST.COM
I20260620 23:33:53.010046 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:52.993171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.0.96.195:33113)
0620 23:33:52.993491 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:52.993497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:52.993513 (+ 16us) server_negotiation.cc:407] Connection header received
0620 23:33:52.993572 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:52.993576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:52.993655 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:52.993777 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:52.995099 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.995982 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.997324 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:52.997899 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:52.998039 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:53.003346 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:53.003362 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:53.003380 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:53.003434 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:53.007034 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:53.007791 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:53.007799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:53.007801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:53.007968 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:53.008367 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:53.008370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:53.008372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:53.008720 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:53.009086 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:53.009390 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:33:53.009648 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260620 23:33:53.010969 20235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:53.150483 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20112
I20260620 23:33:53.167618 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 19751
I20260620 23:33:53.169332 19813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:53.282538 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 19751
2026-06-20T23:33:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4974 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T23:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:53Z Disabled control of system clock
I20260620 23:33:53.367875 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36291
--webserver_interface=127.0.96.254
--webserver_port=0
--builtin_ntp_servers=127.0.96.212:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:33:53.703608 20293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:53.703915 20293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:53.704061 20293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:53.711452 20293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:53.711580 20293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:53.711618 20293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:53.711714 20293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:53.711791 20293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:53.721158 20293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:53.722535 20293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:53.724409 20293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:53.737607 20301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:53.737619 20299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:53.737673 20298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:53.739284 20293 server_base.cc:1034] running on GCE node
I20260620 23:33:53.740111 20293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:53.741708 20293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:53.742969 20293 hybrid_clock.cc:648] HybridClock initialized: now 1781998433742907 us; error 69 us; skew 500 ppm
I20260620 23:33:53.743345 20293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:53.746057 20293 webserver.cc:460] Webserver started at http://127.0.96.254:35361/ using document root <none> and password file <none>
I20260620 23:33:53.746788 20293 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:53.746928 20293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:53.747253 20293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:53.750231 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "63eb38ae1b7e45fe9960293f42cb5094"
format_stamp: "Formatted at 2026-06-20 23:33:53 on dist-test-slave-v34r"
server_key: "b0c3e970f3e388ea1e87cd59957ac23b"
server_key_iv: "e5eba83d119e75208c5b71abc155f6bc"
server_key_version: "encryptionkey@0"
I20260620 23:33:53.751051 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "63eb38ae1b7e45fe9960293f42cb5094"
format_stamp: "Formatted at 2026-06-20 23:33:53 on dist-test-slave-v34r"
server_key: "b0c3e970f3e388ea1e87cd59957ac23b"
server_key_iv: "e5eba83d119e75208c5b71abc155f6bc"
server_key_version: "encryptionkey@0"
I20260620 23:33:53.756691 20293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 23:33:53.760707 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:53.762471 20293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:33:53.762689 20293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "63eb38ae1b7e45fe9960293f42cb5094"
format_stamp: "Formatted at 2026-06-20 23:33:53 on dist-test-slave-v34r"
server_key: "b0c3e970f3e388ea1e87cd59957ac23b"
server_key_iv: "e5eba83d119e75208c5b71abc155f6bc"
server_key_version: "encryptionkey@0"
I20260620 23:33:53.762931 20293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:53.782016 20293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:53.782912 20293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:53.783215 20293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:53.805208 20293 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36291
I20260620 23:33:53.805279 20358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36291 every 8 connection(s)
I20260620 23:33:53.806712 20293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:53.811774 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:53.815178 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20293
I20260620 23:33:53.815402 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:53.815994 387 external_mini_cluster.cc:1428] Setting key 9ae9c35ad9c9a2c034ade773bf50e811
I20260620 23:33:53.827138 20359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094: Bootstrap starting.
I20260620 23:33:53.831962 20359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:53.832626 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:53.818738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.0.96.254:36291)
0620 23:33:53.820184 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:53.820238 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:53.820289 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:33:53.822403 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:53.822432 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:53.823025 (+ 593us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:53.823496 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:53.824154 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:53.825623 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:53.827045 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:53.828023 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:53.828248 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:53.829895 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:53.829903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:53.829918 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:53.830558 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:53.830662 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:53.830685 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:53.830782 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:53.831134 (+ 352us) server_negotiation.cc:299] Negotiation successful
0620 23:33:53.831551 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":532,"threads_started":1}
I20260620 23:33:53.833451 20359 log.cc:824] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:53.836861 20359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094: No bootstrap required, opened a new log
I20260620 23:33:53.845209 20359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eb38ae1b7e45fe9960293f42cb5094" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36291 } }
I20260620 23:33:53.845760 20359 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:53.845881 20359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63eb38ae1b7e45fe9960293f42cb5094, State: Initialized, Role: FOLLOWER
I20260620 23:33:53.846746 20359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eb38ae1b7e45fe9960293f42cb5094" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36291 } }
I20260620 23:33:53.847157 20359 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:53.847334 20359 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:53.847530 20359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:53.850694 20359 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eb38ae1b7e45fe9960293f42cb5094" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36291 } }
I20260620 23:33:53.851487 20359 leader_election.cc:302] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63eb38ae1b7e45fe9960293f42cb5094; no voters:
I20260620 23:33:53.852332 20359 leader_election.cc:288] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:53.852525 20364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:53.854012 20364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [term 1 LEADER]: Becoming Leader. State: Replica: 63eb38ae1b7e45fe9960293f42cb5094, State: Running, Role: LEADER
I20260620 23:33:53.854606 20364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eb38ae1b7e45fe9960293f42cb5094" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36291 } }
I20260620 23:33:53.855301 20359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:53.858299 20366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63eb38ae1b7e45fe9960293f42cb5094. Latest consensus state: current_term: 1 leader_uuid: "63eb38ae1b7e45fe9960293f42cb5094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eb38ae1b7e45fe9960293f42cb5094" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36291 } } }
I20260620 23:33:53.858429 20365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63eb38ae1b7e45fe9960293f42cb5094" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63eb38ae1b7e45fe9960293f42cb5094" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36291 } } }
I20260620 23:33:53.858763 20366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:53.858901 20365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:53.865950 20371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:53.874648 20371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:53.886253 20371 catalog_manager.cc:1347] Generated new cluster ID: 2ca210d8a86e4a429445922b74e0ee08
I20260620 23:33:53.886371 20371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:53.903319 20371 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:53.905356 20371 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:53.916296 20371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094: Generated new TSK 0
I20260620 23:33:53.917637 20371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:33:53.935362 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.193:0
--local_ip_for_outbound_sockets=127.0.96.193
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36291
--builtin_ntp_servers=127.0.96.212:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 23:33:54.323494 20383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:54.323796 20383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:54.323930 20383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:54.331560 20383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:54.331671 20383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:54.331830 20383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.193
I20260620 23:33:54.341465 20383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.96.193
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.193
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:54.342845 20383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:54.344646 20383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:54.359867 20391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:54.360615 20388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:54.360693 20389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:54.361215 20383 server_base.cc:1034] running on GCE node
I20260620 23:33:54.362061 20383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:54.363233 20383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:54.364522 20383 hybrid_clock.cc:648] HybridClock initialized: now 1781998434364435 us; error 57 us; skew 500 ppm
I20260620 23:33:54.364928 20383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:54.368340 20383 webserver.cc:460] Webserver started at http://127.0.96.193:36227/ using document root <none> and password file <none>
I20260620 23:33:54.369412 20383 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:54.369544 20383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:54.369984 20383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:54.373975 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data/instance:
uuid: "9cbe407f901745c28b0f899a18c892d2"
format_stamp: "Formatted at 2026-06-20 23:33:54 on dist-test-slave-v34r"
server_key: "71da51582f28b4103a198f3bb2443ec1"
server_key_iv: "f247ff15e3868bb91fd22ae5116cf2ed"
server_key_version: "encryptionkey@0"
I20260620 23:33:54.375051 20383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal/instance:
uuid: "9cbe407f901745c28b0f899a18c892d2"
format_stamp: "Formatted at 2026-06-20 23:33:54 on dist-test-slave-v34r"
server_key: "71da51582f28b4103a198f3bb2443ec1"
server_key_iv: "f247ff15e3868bb91fd22ae5116cf2ed"
server_key_version: "encryptionkey@0"
I20260620 23:33:54.382089 20383 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:33:54.386893 20397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:54.389022 20383 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260620 23:33:54.389359 20383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal
uuid: "9cbe407f901745c28b0f899a18c892d2"
format_stamp: "Formatted at 2026-06-20 23:33:54 on dist-test-slave-v34r"
server_key: "71da51582f28b4103a198f3bb2443ec1"
server_key_iv: "f247ff15e3868bb91fd22ae5116cf2ed"
server_key_version: "encryptionkey@0"
I20260620 23:33:54.389693 20383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:54.411450 20383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:54.412721 20383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:54.413208 20383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:54.414554 20383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:54.416437 20383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:54.416597 20383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:54.416752 20383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:54.416853 20383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:54.458588 20383 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.193:38843
I20260620 23:33:54.458644 20509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.193:38843 every 8 connection(s)
I20260620 23:33:54.460325 20383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/data/info.pb
I20260620 23:33:54.462641 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20383
I20260620 23:33:54.462853 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-0/wal/instance
I20260620 23:33:54.463423 387 external_mini_cluster.cc:1428] Setting key 5bf07b7205029e3a1033a511986e14eb
I20260620 23:33:54.467445 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.194:0
--local_ip_for_outbound_sockets=127.0.96.194
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36291
--builtin_ntp_servers=127.0.96.212:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:33:54.478804 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:54.463487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.193:56029 (local address 127.0.96.254:36291)
0620 23:33:54.464086 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:54.464095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:54.465686 (+ 1591us) server_negotiation.cc:407] Connection header received
0620 23:33:54.469062 (+ 3376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:54.469068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:54.469154 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:54.469314 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:54.471324 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.472064 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.474993 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.475443 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.475556 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:54.477882 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:54.477887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:54.477891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:54.477914 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:54.477965 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:54.477979 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:54.478067 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:54.478404 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 23:33:54.478494 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":265,"threads_started":1}
I20260620 23:33:54.480505 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:54.464070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36291 (local address 127.0.96.193:56029)
0620 23:33:54.465490 (+ 1420us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:54.465552 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:54.468553 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:54.469718 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:54.469736 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:54.469907 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:54.471003 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:54.471038 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.472275 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.472284 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:54.474792 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:54.474816 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.475803 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.475814 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:54.476230 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:54.477347 (+ 1117us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:54.477354 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:54.477445 (+ 91us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:54.477605 (+ 160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:54.478110 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:54.478125 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:54.478137 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:33:54.478371 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:33:54.478968 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":378,"threads_started":1}
I20260620 23:33:54.482610 20510 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36291
I20260620 23:33:54.483036 20510 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:54.484038 20510 heartbeater.cc:511] Master 127.0.96.254:36291 requested a full tablet report, sending...
I20260620 23:33:54.486745 20324 ts_manager.cc:194] Registered new tserver with Master: 9cbe407f901745c28b0f899a18c892d2 (127.0.96.193:38843)
I20260620 23:33:54.488545 20324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.193:56029
W20260620 23:33:54.812275 20514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:54.812616 20514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:54.812767 20514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:54.820283 20514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:54.820444 20514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:54.820616 20514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.194
I20260620 23:33:54.829943 20514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.96.194
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.194
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:54.831348 20514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:54.833099 20514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:54.846946 20519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:54.847093 20520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:54.847393 20522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:54.848173 20514 server_base.cc:1034] running on GCE node
I20260620 23:33:54.848935 20514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:54.850064 20514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:54.851341 20514 hybrid_clock.cc:648] HybridClock initialized: now 1781998434851280 us; error 59 us; skew 500 ppm
I20260620 23:33:54.851712 20514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:54.854501 20514 webserver.cc:460] Webserver started at http://127.0.96.194:35395/ using document root <none> and password file <none>
I20260620 23:33:54.855309 20514 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:54.855443 20514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:54.855782 20514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:54.858794 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data/instance:
uuid: "4736f3a186474e1c8a53c78ca1ce6714"
format_stamp: "Formatted at 2026-06-20 23:33:54 on dist-test-slave-v34r"
server_key: "d705c35eb1e078428e0c5a9c71504ab3"
server_key_iv: "734df0ab7571cbdf508a24cc7166fd51"
server_key_version: "encryptionkey@0"
I20260620 23:33:54.859591 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal/instance:
uuid: "4736f3a186474e1c8a53c78ca1ce6714"
format_stamp: "Formatted at 2026-06-20 23:33:54 on dist-test-slave-v34r"
server_key: "d705c35eb1e078428e0c5a9c71504ab3"
server_key_iv: "734df0ab7571cbdf508a24cc7166fd51"
server_key_version: "encryptionkey@0"
I20260620 23:33:54.865392 20514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 23:33:54.869423 20528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:54.870959 20514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:33:54.871147 20514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal
uuid: "4736f3a186474e1c8a53c78ca1ce6714"
format_stamp: "Formatted at 2026-06-20 23:33:54 on dist-test-slave-v34r"
server_key: "d705c35eb1e078428e0c5a9c71504ab3"
server_key_iv: "734df0ab7571cbdf508a24cc7166fd51"
server_key_version: "encryptionkey@0"
I20260620 23:33:54.871368 20514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:54.903303 20514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:54.904162 20514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:54.904471 20514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:54.905578 20514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:54.907178 20514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:54.907310 20514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:54.907439 20514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:54.907534 20514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:54.945521 20514 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.194:34557
I20260620 23:33:54.945627 20640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.194:34557 every 8 connection(s)
I20260620 23:33:54.946957 20514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/data/info.pb
I20260620 23:33:54.953742 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20514
I20260620 23:33:54.953977 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal/instance
I20260620 23:33:54.954599 387 external_mini_cluster.cc:1428] Setting key fd2fe9749bca5268a42670b65b7a6099
I20260620 23:33:54.958431 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.96.195:0
--local_ip_for_outbound_sockets=127.0.96.195
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36291
--builtin_ntp_servers=127.0.96.212:45695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 23:33:54.965167 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:54.950750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.194:42635 (local address 127.0.96.254:36291)
0620 23:33:54.951022 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:54.951028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:54.952635 (+ 1607us) server_negotiation.cc:407] Connection header received
0620 23:33:54.955734 (+ 3099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:54.955740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:54.955829 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:54.955937 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:54.957741 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.958466 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.961590 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.962009 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.962120 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:54.964199 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:54.964204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:54.964207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:54.964228 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:54.964271 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:54.964283 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:54.964365 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:54.964753 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 23:33:54.964838 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:33:54.967315 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:54.951249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36291 (local address 127.0.96.194:42635)
0620 23:33:54.952384 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:54.952458 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:54.955170 (+ 2712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:54.956270 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:54.956281 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:54.956408 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:54.957493 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:54.957525 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.958705 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.958715 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:54.961351 (+ 2636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:54.961380 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:54.962309 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:54.962319 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:54.962720 (+ 401us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:54.963697 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:54.963702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:54.963777 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:54.963925 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:54.964400 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:54.964416 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:54.964428 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 23:33:54.964688 (+ 260us) client_negotiation.cc:240] Negotiation successful
0620 23:33:54.965306 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":342,"threads_started":1}
I20260620 23:33:54.969514 20641 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36291
I20260620 23:33:54.969975 20641 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:54.971208 20641 heartbeater.cc:511] Master 127.0.96.254:36291 requested a full tablet report, sending...
I20260620 23:33:54.973354 20324 ts_manager.cc:194] Registered new tserver with Master: 4736f3a186474e1c8a53c78ca1ce6714 (127.0.96.194:34557)
I20260620 23:33:54.974676 20324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.194:42635
W20260620 23:33:55.300038 20644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:55.300375 20644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:55.300522 20644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:55.308157 20644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:55.308293 20644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:55.308465 20644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.96.195
I20260620 23:33:55.317943 20644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:45695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.96.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.96.195
--webserver_port=0
--tserver_master_addrs=127.0.96.254:36291
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.96.195
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:55.319350 20644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:55.321195 20644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:55.337059 20649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:55.337083 20652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:55.337177 20650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:55.343317 20644 server_base.cc:1034] running on GCE node
I20260620 23:33:55.344327 20644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:55.345763 20644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:55.347052 20644 hybrid_clock.cc:648] HybridClock initialized: now 1781998435346971 us; error 59 us; skew 500 ppm
I20260620 23:33:55.347561 20644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 23:33:55.350656 20644 webserver.cc:460] Webserver started at http://127.0.96.195:36733/ using document root <none> and password file <none>
I20260620 23:33:55.351611 20644 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:55.351727 20644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:55.352142 20644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:55.356084 20644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data/instance:
uuid: "f52defb8370e456bb5baa2703b7463c3"
format_stamp: "Formatted at 2026-06-20 23:33:55 on dist-test-slave-v34r"
server_key: "02f2781a4a3537146574b4690c6c03bb"
server_key_iv: "16f88cf5bab802ab468289f8294d8ca9"
server_key_version: "encryptionkey@0"
I20260620 23:33:55.357512 20644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal/instance:
uuid: "f52defb8370e456bb5baa2703b7463c3"
format_stamp: "Formatted at 2026-06-20 23:33:55 on dist-test-slave-v34r"
server_key: "02f2781a4a3537146574b4690c6c03bb"
server_key_iv: "16f88cf5bab802ab468289f8294d8ca9"
server_key_version: "encryptionkey@0"
I20260620 23:33:55.364817 20644 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260620 23:33:55.369758 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:55.371857 20644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260620 23:33:55.372169 20644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal
uuid: "f52defb8370e456bb5baa2703b7463c3"
format_stamp: "Formatted at 2026-06-20 23:33:55 on dist-test-slave-v34r"
server_key: "02f2781a4a3537146574b4690c6c03bb"
server_key_iv: "16f88cf5bab802ab468289f8294d8ca9"
server_key_version: "encryptionkey@0"
I20260620 23:33:55.372525 20644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:55.391531 20644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:55.392589 20644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:55.393033 20644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:55.394462 20644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 23:33:55.396442 20644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 23:33:55.396543 20644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:55.396735 20644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 23:33:55.396816 20644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:55.438534 20644 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.195:38397
I20260620 23:33:55.438654 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.195:38397 every 8 connection(s)
I20260620 23:33:55.440003 20644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/data/info.pb
I20260620 23:33:55.444152 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20644
I20260620 23:33:55.444371 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal/instance
I20260620 23:33:55.444895 387 external_mini_cluster.cc:1428] Setting key 28d85230601f1d3e4f5e9e4326462991
I20260620 23:33:55.455983 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.443470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:44305 (local address 127.0.96.254:36291)
0620 23:33:55.443824 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.443834 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.445590 (+ 1756us) server_negotiation.cc:407] Connection header received
0620 23:33:55.447842 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.447848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.447942 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.448047 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:55.449696 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.450493 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.452458 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.452857 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.452963 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.454985 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:55.454990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:55.454993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:55.455014 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:55.455047 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:55.455059 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:55.455139 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.455545 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.455629 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 23:33:55.457587 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.444031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:36291 (local address 127.0.96.195:44305)
0620 23:33:55.445397 (+ 1366us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:55.445448 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:55.447557 (+ 2109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:55.448462 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:55.448474 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:55.448605 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:33:55.449476 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.449500 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.450713 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.450719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.452271 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.452290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.453287 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.453298 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.453573 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.454562 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 23:33:55.454565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:33:55.454636 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 23:33:55.454758 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:33:55.455268 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:33:55.455279 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:33:55.455287 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 23:33:55.455490 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 23:33:55.456018 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":501,"threads_started":1}
I20260620 23:33:55.459589 20771 heartbeater.cc:347] Connected to a master server at 127.0.96.254:36291
I20260620 23:33:55.460098 20771 heartbeater.cc:464] Registering TS with master...
I20260620 23:33:55.461481 20771 heartbeater.cc:511] Master 127.0.96.254:36291 requested a full tablet report, sending...
I20260620 23:33:55.463238 20324 ts_manager.cc:194] Registered new tserver with Master: f52defb8370e456bb5baa2703b7463c3 (127.0.96.195:38397)
I20260620 23:33:55.464340 20324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.96.195:44305
I20260620 23:33:55.475309 387 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 23:33:55.488083 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.480723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47474 (local address 127.0.96.254:36291)
0620 23:33:55.480969 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.480974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.481504 (+ 530us) server_negotiation.cc:407] Connection header received
0620 23:33:55.481659 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.481664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.481743 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.481862 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:55.482519 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.483266 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.485221 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.485626 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.485790 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.487294 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:55.487300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:55.487303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:55.487323 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:55.487374 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:55.487387 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:55.487473 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.487750 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.487838 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:33:55.492745 20510 heartbeater.cc:503] Master 127.0.96.254:36291 was elected leader, sending a full tablet report...
I20260620 23:33:55.493211 20324 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 23:33:55.501184 20324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 23:33:55.530274 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.519419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:46620)
0620 23:33:55.520279 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:55.520311 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:55.520536 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:55.522104 (+ 1568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:55.522110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:55.522138 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:55.522843 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.522857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.524858 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.524863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.527202 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.527219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.529319 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.529323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.529470 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.529503 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 23:33:55.529664 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 23:33:55.529866 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":197,"thread_start_us":933,"threads_started":1}
I20260620 23:33:55.530251 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.520157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.0.96.193:38843)
0620 23:33:55.521337 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.521352 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.521395 (+ 43us) server_negotiation.cc:407] Connection header received
0620 23:33:55.521530 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.521538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.521842 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.522108 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:55.523046 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.524699 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.527441 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.528740 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.528877 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.529205 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.529718 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.529884 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":667,"threads_started":1}
I20260620 23:33:55.531469 20786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.517890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34557 (local address 127.0.0.1:43872)
0620 23:33:55.518936 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:55.518955 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:55.519058 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:55.521788 (+ 2730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:55.521796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:55.521829 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:55.522389 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.522405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.524939 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.524945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.527986 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.528011 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.530652 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.530658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.530863 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.530899 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:33:55.531012 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 23:33:55.531149 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":127,"thread_start_us":743,"threads_started":1}
I20260620 23:33:55.531919 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.517548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.0.96.194:34557)
0620 23:33:55.520932 (+ 3384us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.520942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.520974 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:33:55.521116 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.521126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.521511 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.521747 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:55.522744 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.524740 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.528351 (+ 3611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.530068 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.530260 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.530532 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.531091 (+ 559us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.531319 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1844,"thread_start_us":1721,"threads_started":1}
I20260620 23:33:55.532569 20783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.516708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.195:38397 (local address 127.0.0.1:53284)
0620 23:33:55.518124 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:55.518175 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:55.518343 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:55.522527 (+ 4184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:55.522536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:55.522572 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:55.523167 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.523179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.525315 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.525321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.528006 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.528015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.531610 (+ 3595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.531618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.531839 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.531875 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 23:33:55.531994 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 23:33:55.532174 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1110,"mutex_wait_us":52,"thread_start_us":959,"threads_started":1}
I20260620 23:33:55.532750 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.516907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.0.96.195:38397)
0620 23:33:55.521767 (+ 4860us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.521775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.521810 (+ 35us) server_negotiation.cc:407] Connection header received
0620 23:33:55.521914 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.521921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.522232 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.522495 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:55.523568 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.525099 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.528351 (+ 3252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.530964 (+ 2613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.531163 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.531452 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.532144 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.532376 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4653,"thread_start_us":4474,"threads_started":1}
I20260620 23:33:55.536679 20706 tablet_service.cc:1467] Processing CreateTablet for tablet 41ab28c766604f3cb55d73adc8fcacea (DEFAULT_TABLE table=test-table [id=1786ef47e0364fa3b242794442445746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:55.536645 20445 tablet_service.cc:1467] Processing CreateTablet for tablet 41ab28c766604f3cb55d73adc8fcacea (DEFAULT_TABLE table=test-table [id=1786ef47e0364fa3b242794442445746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:55.536650 20576 tablet_service.cc:1467] Processing CreateTablet for tablet 41ab28c766604f3cb55d73adc8fcacea (DEFAULT_TABLE table=test-table [id=1786ef47e0364fa3b242794442445746]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 23:33:55.538200 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ab28c766604f3cb55d73adc8fcacea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:55.538535 20445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ab28c766604f3cb55d73adc8fcacea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:55.538535 20576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ab28c766604f3cb55d73adc8fcacea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:33:55.550170 20789 tablet_bootstrap.cc:492] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Bootstrap starting.
I20260620 23:33:55.553577 20789 tablet_bootstrap.cc:654] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:55.554581 20789 log.cc:824] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:55.558413 20792 tablet_bootstrap.cc:492] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Bootstrap starting.
I20260620 23:33:55.558487 20789 tablet_bootstrap.cc:492] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: No bootstrap required, opened a new log
I20260620 23:33:55.558739 20789 ts_tablet_manager.cc:1397] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 23:33:55.562165 20792 tablet_bootstrap.cc:654] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:55.563412 20792 log.cc:824] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:55.564441 20791 tablet_bootstrap.cc:492] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Bootstrap starting.
I20260620 23:33:55.567728 20792 tablet_bootstrap.cc:492] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: No bootstrap required, opened a new log
I20260620 23:33:55.568094 20792 ts_tablet_manager.cc:1397] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 23:33:55.569149 20791 tablet_bootstrap.cc:654] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:55.569393 20789 raft_consensus.cc:348] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.569911 20789 raft_consensus.cc:374] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:55.570020 20789 raft_consensus.cc:729] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f52defb8370e456bb5baa2703b7463c3, State: Initialized, Role: FOLLOWER
I20260620 23:33:55.570469 20791 log.cc:824] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:55.570703 20789 consensus_queue.cc:260] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.574337 20771 heartbeater.cc:503] Master 127.0.96.254:36291 was elected leader, sending a full tablet report...
I20260620 23:33:55.575043 20789 ts_tablet_manager.cc:1428] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Time spent starting tablet: real 0.016s user 0.010s sys 0.008s
I20260620 23:33:55.576589 20791 tablet_bootstrap.cc:492] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: No bootstrap required, opened a new log
I20260620 23:33:55.576972 20791 ts_tablet_manager.cc:1397] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260620 23:33:55.583844 20792 raft_consensus.cc:348] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.584336 20792 raft_consensus.cc:374] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:55.584457 20792 raft_consensus.cc:729] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbe407f901745c28b0f899a18c892d2, State: Initialized, Role: FOLLOWER
I20260620 23:33:55.585311 20792 consensus_queue.cc:260] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.585659 20791 raft_consensus.cc:348] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.586108 20791 raft_consensus.cc:374] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:55.586232 20791 raft_consensus.cc:729] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4736f3a186474e1c8a53c78ca1ce6714, State: Initialized, Role: FOLLOWER
I20260620 23:33:55.586962 20791 consensus_queue.cc:260] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.587522 20792 ts_tablet_manager.cc:1428] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Time spent starting tablet: real 0.019s user 0.009s sys 0.004s
I20260620 23:33:55.588459 20641 heartbeater.cc:503] Master 127.0.96.254:36291 was elected leader, sending a full tablet report...
I20260620 23:33:55.589172 20791 ts_tablet_manager.cc:1428] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
W20260620 23:33:55.692363 20772 tablet.cc:2369] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:55.699575 20642 tablet.cc:2369] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 23:33:55.713824 20511 tablet.cc:2369] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:33:55.894402 20795 raft_consensus.cc:482] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 23:33:55.894770 20795 raft_consensus.cc:504] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.896504 20795 leader_election.cc:288] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4736f3a186474e1c8a53c78ca1ce6714 (127.0.96.194:34557), 9cbe407f901745c28b0f899a18c892d2 (127.0.96.193:38843)
I20260620 23:33:55.904457 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.897288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.96.195:52231)
0620 23:33:55.897564 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:55.897588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:55.897798 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:55.898381 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:55.898385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:55.898454 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:55.898695 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.898707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.900128 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.900134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.902033 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.902045 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.903595 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.903601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.903812 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.903857 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 23:33:55.903989 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 23:33:55.904148 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":5}
I20260620 23:33:55.904561 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.897107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:52231 (local address 127.0.96.193:38843)
0620 23:33:55.897379 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.897386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.897750 (+ 364us) server_negotiation.cc:407] Connection header received
0620 23:33:55.898021 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.898027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.898145 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.898299 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:55.898897 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.899959 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.902252 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.903389 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.903545 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.903608 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.904074 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.904240 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 23:33:55.905077 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.897249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.96.195:53679 (local address 127.0.96.194:34557)
0620 23:33:55.897567 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.897575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.898319 (+ 744us) server_negotiation.cc:407] Connection header received
0620 23:33:55.898488 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.898495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.898589 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.898820 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 23:33:55.899591 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.900509 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.902939 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.903943 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.904094 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.904160 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.904603 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.904761 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 23:33:55.905196 20798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.897285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.194:34557 (local address 127.0.96.195:53679)
0620 23:33:55.898173 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:55.898197 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:33:55.898334 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:33:55.898897 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:33:55.898902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:33:55.898934 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 23:33:55.899377 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.899391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.900706 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.900711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.902688 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:33:55.902711 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.904161 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.904168 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:33:55.904386 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.904424 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 23:33:55.904558 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 23:33:55.904729 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":145,"thread_start_us":333,"threads_started":1}
I20260620 23:33:55.906173 20465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ab28c766604f3cb55d73adc8fcacea" candidate_uuid: "f52defb8370e456bb5baa2703b7463c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbe407f901745c28b0f899a18c892d2" is_pre_election: true
I20260620 23:33:55.906204 20596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ab28c766604f3cb55d73adc8fcacea" candidate_uuid: "f52defb8370e456bb5baa2703b7463c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4736f3a186474e1c8a53c78ca1ce6714" is_pre_election: true
I20260620 23:33:55.906730 20596 raft_consensus.cc:2454] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f52defb8370e456bb5baa2703b7463c3 in term 0.
I20260620 23:33:55.906805 20465 raft_consensus.cc:2454] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f52defb8370e456bb5baa2703b7463c3 in term 0.
I20260620 23:33:55.907496 20659 leader_election.cc:302] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4736f3a186474e1c8a53c78ca1ce6714, f52defb8370e456bb5baa2703b7463c3; no voters:
I20260620 23:33:55.907928 20795 raft_consensus.cc:2781] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 23:33:55.908084 20795 raft_consensus.cc:482] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 23:33:55.908229 20795 raft_consensus.cc:3037] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:55.910455 20795 raft_consensus.cc:504] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.911109 20795 leader_election.cc:288] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4736f3a186474e1c8a53c78ca1ce6714 (127.0.96.194:34557), 9cbe407f901745c28b0f899a18c892d2 (127.0.96.193:38843)
I20260620 23:33:55.911666 20596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ab28c766604f3cb55d73adc8fcacea" candidate_uuid: "f52defb8370e456bb5baa2703b7463c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4736f3a186474e1c8a53c78ca1ce6714"
I20260620 23:33:55.911890 20596 raft_consensus.cc:3037] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:55.911921 20465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ab28c766604f3cb55d73adc8fcacea" candidate_uuid: "f52defb8370e456bb5baa2703b7463c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cbe407f901745c28b0f899a18c892d2"
I20260620 23:33:55.912249 20465 raft_consensus.cc:3037] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:55.914405 20596 raft_consensus.cc:2454] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f52defb8370e456bb5baa2703b7463c3 in term 1.
I20260620 23:33:55.914896 20659 leader_election.cc:302] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4736f3a186474e1c8a53c78ca1ce6714, f52defb8370e456bb5baa2703b7463c3; no voters:
I20260620 23:33:55.915273 20795 raft_consensus.cc:2781] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:55.915306 20465 raft_consensus.cc:2454] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f52defb8370e456bb5baa2703b7463c3 in term 1.
I20260620 23:33:55.915978 20795 raft_consensus.cc:686] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 1 LEADER]: Becoming Leader. State: Replica: f52defb8370e456bb5baa2703b7463c3, State: Running, Role: LEADER
I20260620 23:33:55.916615 20795 consensus_queue.cc:237] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } }
I20260620 23:33:55.922178 20324 catalog_manager.cc:5495] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to f52defb8370e456bb5baa2703b7463c3 (127.0.96.195). New cstate: current_term: 1 leader_uuid: "f52defb8370e456bb5baa2703b7463c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f52defb8370e456bb5baa2703b7463c3" member_type: VOTER last_known_addr { host: "127.0.96.195" port: 38397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 } health_report { overall_health: UNKNOWN } } }
I20260620 23:33:55.990480 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:55.984354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.0.96.195:38397)
0620 23:33:55.984564 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:55.984569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:55.984589 (+ 20us) server_negotiation.cc:407] Connection header received
0620 23:33:55.984729 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:55.984734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:55.984830 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:55.985014 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:33:55.985460 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.986343 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.987853 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:55.988331 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:55.988450 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:55.989161 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:33:55.989858 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:33:55.990000 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:55.990121 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 23:33:55.990228 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:33:55.999997 20596 raft_consensus.cc:1261] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 1 FOLLOWER]: Refusing update from remote peer f52defb8370e456bb5baa2703b7463c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:56.000000 20465 raft_consensus.cc:1261] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 1 FOLLOWER]: Refusing update from remote peer f52defb8370e456bb5baa2703b7463c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:33:56.001837 20799 consensus_queue.cc:1035] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cbe407f901745c28b0f899a18c892d2" member_type: VOTER last_known_addr { host: "127.0.96.193" port: 38843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:56.002439 20795 consensus_queue.cc:1035] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4736f3a186474e1c8a53c78ca1ce6714" member_type: VOTER last_known_addr { host: "127.0.96.194" port: 34557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:33:56.018138 20802 mvcc.cc:205] Tried to move back new op lower bound from 7299065593842905088 to 7299065593519730688. Current Snapshot: MvccSnapshot[applied={T|T < 7299065593842905088}]
I20260620 23:33:56.024878 20804 mvcc.cc:205] Tried to move back new op lower bound from 7299065593842905088 to 7299065593519730688. Current Snapshot: MvccSnapshot[applied={T|T < 7299065593842905088}]
I20260620 23:33:56.044073 20323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:56.044411 20323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 23:33:56.049013 20323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63eb38ae1b7e45fe9960293f42cb5094: Sending DeleteTablet for 3 replicas of tablet 41ab28c766604f3cb55d73adc8fcacea
I20260620 23:33:56.050611 20445 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ab28c766604f3cb55d73adc8fcacea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:56 UTC) from {username='slave'} at 127.0.0.1:46620
I20260620 23:33:56.050704 20706 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ab28c766604f3cb55d73adc8fcacea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:56 UTC) from {username='slave'} at 127.0.0.1:53284
I20260620 23:33:56.050697 20576 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ab28c766604f3cb55d73adc8fcacea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 23:33:56 UTC) from {username='slave'} at 127.0.0.1:43872
I20260620 23:33:56.057111 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20383
I20260620 23:33:56.062042 20813 tablet_replica.cc:330] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: stopping tablet replica
I20260620 23:33:56.062279 20814 tablet_replica.cc:330] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: stopping tablet replica
I20260620 23:33:56.062902 20814 raft_consensus.cc:2229] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:56.062913 20813 raft_consensus.cc:2229] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 23:33:56.063643 20814 raft_consensus.cc:2258] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:56.063917 20813 raft_consensus.cc:2258] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:56.065765 20812 tablet_replica.cc:330] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: stopping tablet replica
I20260620 23:33:56.066459 20812 raft_consensus.cc:2229] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 23:33:56.067250 20812 raft_consensus.cc:2258] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 23:33:56.068882 20813 ts_tablet_manager.cc:1905] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:56.069906 20814 ts_tablet_manager.cc:1905] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:56.070400 20812 ts_tablet_manager.cc:1905] T 41ab28c766604f3cb55d73adc8fcacea P 9cbe407f901745c28b0f899a18c892d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 23:33:56.077677 20788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.058828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.0.96.193:38843)
0620 23:33:56.059029 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:56.059035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:56.064855 (+ 5820us) server_negotiation.cc:407] Connection header received
0620 23:33:56.065170 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:56.065177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:56.065378 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:56.065520 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:56.066292 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:56.067562 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:56.069554 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:56.070187 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:56.070302 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:56.074790 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:56.074803 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:56.074823 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:56.076124 (+ 1301us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:56.076196 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:56.076224 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:56.076386 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:56.077060 (+ 674us) server_negotiation.cc:299] Negotiation successful
0620 23:33:56.077184 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:33:56.078627 20505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:56.078608 20813 ts_tablet_manager.cc:1918] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:56.078971 20813 log.cc:1196] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-2/wal/wals/41ab28c766604f3cb55d73adc8fcacea
I20260620 23:33:56.079749 20813 ts_tablet_manager.cc:1939] T 41ab28c766604f3cb55d73adc8fcacea P f52defb8370e456bb5baa2703b7463c3: Deleting consensus metadata
I20260620 23:33:56.082228 20814 ts_tablet_manager.cc:1918] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 23:33:56.082450 20814 log.cc:1196] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Deleting WAL directory at /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781998087756853-387-0/minicluster-data/ts-1/wal/wals/41ab28c766604f3cb55d73adc8fcacea
I20260620 23:33:56.083024 20814 ts_tablet_manager.cc:1939] T 41ab28c766604f3cb55d73adc8fcacea P 4736f3a186474e1c8a53c78ca1ce6714: Deleting consensus metadata
I20260620 23:33:56.082872 20311 catalog_manager.cc:4841] TS f52defb8370e456bb5baa2703b7463c3 (127.0.96.195:38397): tablet 41ab28c766604f3cb55d73adc8fcacea (table test-table [id=1786ef47e0364fa3b242794442445746]) successfully deleted
I20260620 23:33:56.085219 20308 catalog_manager.cc:4841] TS 4736f3a186474e1c8a53c78ca1ce6714 (127.0.96.194:34557): tablet 41ab28c766604f3cb55d73adc8fcacea (table test-table [id=1786ef47e0364fa3b242794442445746]) successfully deleted
I20260620 23:33:56.202648 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20383
W20260620 23:33:56.225069 20310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.96.193:38843 (error 108)
I20260620 23:33:56.225636 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20514
I20260620 23:33:56.227982 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.226653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:46648)
0620 23:33:56.227381 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.227550 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":425,"threads_started":1}
W20260620 23:33:56.228487 20310 catalog_manager.cc:4584] TS 9cbe407f901745c28b0f899a18c892d2 (127.0.96.193:38843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41ab28c766604f3cb55d73adc8fcacea: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
I20260620 23:33:56.235025 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.226682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43886 (local address 127.0.96.194:34557)
0620 23:33:56.226910 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:56.226919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:56.227238 (+ 319us) server_negotiation.cc:407] Connection header received
0620 23:33:56.227329 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:56.227335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:56.227469 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:56.227605 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:56.228282 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:56.229506 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:56.231047 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:56.231703 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:56.231847 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:56.233229 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:56.233239 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:56.233261 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:56.234065 (+ 804us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:56.234131 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:56.234171 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:56.234300 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:56.234496 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:33:56.234631 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 23:33:56.235977 20636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:56.283010 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.282426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:46652)
0620 23:33:56.282730 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.282834 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260620 23:33:56.284341 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.283913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:46654)
0620 23:33:56.284110 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.284187 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 23:33:56.345459 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.344730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:46662)
0620 23:33:56.345087 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.345255 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 23:33:56.347100 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.346540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:46666)
0620 23:33:56.346742 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.346859 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:33:56.362717 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20514
I20260620 23:33:56.385615 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20644
I20260620 23:33:56.395115 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.386674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.96.195:38397)
0620 23:33:56.386903 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:56.386907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:56.386988 (+ 81us) server_negotiation.cc:407] Connection header received
0620 23:33:56.387251 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:56.387256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:56.387391 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:56.387507 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:56.388083 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:56.388970 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:56.390917 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:56.391516 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:56.391649 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:56.393154 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:56.393165 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:56.393185 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:33:56.394017 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:56.394084 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:33:56.394109 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:56.394223 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:56.394464 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 23:33:56.394595 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 23:33:56.395982 20766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:56.428962 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.428122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:58914)
0620 23:33:56.428521 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.428706 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 23:33:56.430794 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.430092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:58930)
0620 23:33:56.430430 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.430572 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 23:33:56.525270 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20644
I20260620 23:33:56.550124 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20293
I20260620 23:33:56.551488 20354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:56.661517 20818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:56.660850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.193:38843 (local address 127.0.0.1:58940)
0620 23:33:56.661127 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 23:33:56.661249 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.193:38843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 23:33:56.661791 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20293
2026-06-20T23:33:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3386 ms)
[----------] 6 tests from EncryptionPolicyTest (24201 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:33:56 dist-test-slave-v34r krb5kdc[20824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:33:56 dist-test-slave-v34r krb5kdc[20824](info): set up 2 sockets
Jun 20 23:33:56 dist-test-slave-v34r krb5kdc[20824](info): commencing operation
krb5kdc: starting...
W20260620 23:33:57.805595 387 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:33:58 dist-test-slave-v34r krb5kdc[20824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:33:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:33:58.119444 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:33:58.481997 20837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:33:58.482311 20837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:33:58.482406 20837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:33:58.489754 20837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:33:58.489881 20837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:33:58.489930 20837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:33:58.489985 20837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:33:58.490023 20837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:33:58.500057 20837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:33:58.501377 20837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:33:58.502624 20837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:33:58.516027 20842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:58.516228 20845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:33:58.516433 20843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:33:58.517388 20837 server_base.cc:1034] running on GCE node
I20260620 23:33:58.518534 20837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:33:58.520638 20837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:33:58.521921 20837 hybrid_clock.cc:648] HybridClock initialized: now 1781998438521815 us; error 93 us; skew 500 ppm
I20260620 23:33:58.522390 20837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:33:58 dist-test-slave-v34r krb5kdc[20824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:33:58.533258 20837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:33:58.535529 20837 webserver.cc:460] Webserver started at http://127.0.0.1:43759/ using document root <none> and password file <none>
I20260620 23:33:58.536458 20837 fs_manager.cc:359] Metadata directory not provided
I20260620 23:33:58.536603 20837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:33:58.536942 20837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:33:58.541277 20837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "41a1c77f0d6f4881b93c75827ba58cf9"
format_stamp: "Formatted at 2026-06-20 23:33:58 on dist-test-slave-v34r"
server_key: "36b56a6add8d2dec187d5023d97b6860"
server_key_iv: "bb88ab80bfacb0002c3126035704b036"
server_key_version: "encryptionkey@0"
I20260620 23:33:58.542265 20837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "41a1c77f0d6f4881b93c75827ba58cf9"
format_stamp: "Formatted at 2026-06-20 23:33:58 on dist-test-slave-v34r"
server_key: "36b56a6add8d2dec187d5023d97b6860"
server_key_iv: "bb88ab80bfacb0002c3126035704b036"
server_key_version: "encryptionkey@0"
I20260620 23:33:58.549252 20837 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:33:58.553817 20852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:33:58.555641 20837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:33:58.555910 20837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "41a1c77f0d6f4881b93c75827ba58cf9"
format_stamp: "Formatted at 2026-06-20 23:33:58 on dist-test-slave-v34r"
server_key: "36b56a6add8d2dec187d5023d97b6860"
server_key_iv: "bb88ab80bfacb0002c3126035704b036"
server_key_version: "encryptionkey@0"
I20260620 23:33:58.556172 20837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:33:58.589054 20837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:33:58.594853 20837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:33:58.595259 20837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:33:58.619222 20837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45411
I20260620 23:33:58.619359 20903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45411 every 8 connection(s)
I20260620 23:33:58.621160 20837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:33:58.622221 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20837
I20260620 23:33:58.622529 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:33:58.623329 387 external_mini_cluster.cc:1428] Setting key 1c9f4040f7a707c632577a09f351424a
I20260620 23:33:58.626133 20904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 23:33:58 dist-test-slave-v34r krb5kdc[20824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:33:58.640720 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9: Bootstrap starting.
I20260620 23:33:58.644311 20904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9: Neither blocks nor log segments found. Creating new log.
I20260620 23:33:58.645539 20904 log.cc:824] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:33:58.648514 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9: No bootstrap required, opened a new log
I20260620 23:33:58.651654 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:58.625791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35551 (local address 127.0.0.1:45411)
0620 23:33:58.626829 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:58.626844 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:58.626898 (+ 54us) server_negotiation.cc:407] Connection header received
0620 23:33:58.628805 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:58.628831 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:58.629445 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:58.629871 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:58.631667 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:58.633340 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:58.635228 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:58.636274 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:58.636498 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:58.642284 (+ 5786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:58.642295 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:58.642320 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:58.642391 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:58.646369 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:58.647457 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:58.647466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:58.647474 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:58.647686 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:58.648200 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:58.648213 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:58.648215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:58.648923 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:58.649407 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:58.649913 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 23:33:58.650336 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":468,"threads_started":1}
I20260620 23:33:58.656872 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:33:58.657307 20904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45411 } }
I20260620 23:33:58.657709 20904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:33:58.657840 20904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a1c77f0d6f4881b93c75827ba58cf9, State: Initialized, Role: FOLLOWER
I20260620 23:33:58.658448 20904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45411 } }
I20260620 23:33:58.658687 20904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:33:58.658831 20904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:33:58.659013 20904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:33:58.661597 20904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45411 } }
I20260620 23:33:58.662139 20904 leader_election.cc:302] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a1c77f0d6f4881b93c75827ba58cf9; no voters:
I20260620 23:33:58.662900 20904 leader_election.cc:288] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:33:58.663107 20913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:33:58.663887 20913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 41a1c77f0d6f4881b93c75827ba58cf9, State: Running, Role: LEADER
I20260620 23:33:58.664609 20913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45411 } }
I20260620 23:33:58.666972 20913 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a1c77f0d6f4881b93c75827ba58cf9. Latest consensus state: current_term: 1 leader_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45411 } } }
I20260620 23:33:58.667342 20913 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:58.667801 20916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a1c77f0d6f4881b93c75827ba58cf9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45411 } } }
I20260620 23:33:58.668244 20916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:33:58.669094 20904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:33:58.677208 20918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:33:58.684582 20918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:33:58.690491 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:33:58.663068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40459 (local address 127.0.0.1:45411)
0620 23:33:58.663302 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 23:33:58.663309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:33:58.666736 (+ 3427us) server_negotiation.cc:407] Connection header received
0620 23:33:58.669372 (+ 2636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:33:58.669378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:33:58.669467 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:33:58.669538 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:33:58.672048 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:58.673741 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:58.676154 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:33:58.677161 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:33:58.677352 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:33:58.681451 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:33:58.681460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:33:58.681464 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:33:58.681517 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:33:58.686328 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:58.687486 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:58.687494 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:58.687498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:58.687728 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:33:58.688336 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:33:58.688341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:33:58.688344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:33:58.688761 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:33:58.689093 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 23:33:58.689859 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 23:33:58.690061 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 23:33:58.698817 20918 catalog_manager.cc:1347] Generated new cluster ID: 155880a3f35946b9b2c6c7a761e6453c
I20260620 23:33:58.698987 20918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:33:58.722390 20918 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:33:58.724648 20918 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:33:58.738729 20918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a1c77f0d6f4881b93c75827ba58cf9: Generated new TSK 0
I20260620 23:33:58.740335 20918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:33:58.751143 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20837
I20260620 23:33:58.752784 20899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:33:58.847072 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20837
W20260620 23:33:58.868860 20822 connection.cc:535] client connection to 127.0.0.1:45411 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45411): Connection reset by peer (error 104)
2026-06-20T23:33:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2175 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:33:58 dist-test-slave-v34r krb5kdc[20936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:33:58 dist-test-slave-v34r krb5kdc[20936](info): set up 2 sockets
Jun 20 23:33:58 dist-test-slave-v34r krb5kdc[20936](info): commencing operation
krb5kdc: starting...
W20260620 23:33:59.981611 387 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:00 dist-test-slave-v34r krb5kdc[20936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:00.288431 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:00.634645 20949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:00.635013 20949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:00.635196 20949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:00.643162 20949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:00.643352 20949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:00.643476 20949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:00.643592 20949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:00.643683 20949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:00.654232 20949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:00.655594 20949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:00.656891 20949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:00.669938 20954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:00.669953 20955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:00.670240 20957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:00.671581 20949 server_base.cc:1034] running on GCE node
I20260620 23:34:00.672659 20949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:00.674567 20949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:00.675848 20949 hybrid_clock.cc:648] HybridClock initialized: now 1781998440675816 us; error 69 us; skew 500 ppm
I20260620 23:34:00.676316 20949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:00 dist-test-slave-v34r krb5kdc[20936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:00.680708 20949 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:00.682951 20949 webserver.cc:460] Webserver started at http://127.0.0.1:40623/ using document root <none> and password file <none>
I20260620 23:34:00.683851 20949 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:00.683976 20949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:00.684296 20949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:00.688102 20949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "da4293fae8c54a56b0a8330e896e3e2a"
format_stamp: "Formatted at 2026-06-20 23:34:00 on dist-test-slave-v34r"
server_key: "905e95e65263fd44887c5ab7d2c2674d"
server_key_iv: "644cd22d212d54e7e209399e567762e5"
server_key_version: "encryptionkey@0"
I20260620 23:34:00.689224 20949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "da4293fae8c54a56b0a8330e896e3e2a"
format_stamp: "Formatted at 2026-06-20 23:34:00 on dist-test-slave-v34r"
server_key: "905e95e65263fd44887c5ab7d2c2674d"
server_key_iv: "644cd22d212d54e7e209399e567762e5"
server_key_version: "encryptionkey@0"
I20260620 23:34:00.696251 20949 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 23:34:00.700934 20964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:00.702833 20949 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:00.703047 20949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "da4293fae8c54a56b0a8330e896e3e2a"
format_stamp: "Formatted at 2026-06-20 23:34:00 on dist-test-slave-v34r"
server_key: "905e95e65263fd44887c5ab7d2c2674d"
server_key_iv: "644cd22d212d54e7e209399e567762e5"
server_key_version: "encryptionkey@0"
I20260620 23:34:00.703271 20949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:00.721367 20949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:00.726761 20949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:00.727078 20949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:00.750121 20949 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40453
I20260620 23:34:00.750121 21015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40453 every 8 connection(s)
I20260620 23:34:00.751971 20949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:00.756245 21016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:00.760630 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 20949
I20260620 23:34:00.760887 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:00.761463 387 external_mini_cluster.cc:1428] Setting key ba74bfcc7849d76ea256709df8e84d67
I20260620 23:34:00.768028 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a: Bootstrap starting.
I20260620 23:34:00.771790 21016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:00.773084 21016 log.cc:824] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:00.776952 21016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a: No bootstrap required, opened a new log
I20260620 23:34:00.778402 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:00.763935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51275 (local address 127.0.0.1:40453)
0620 23:34:00.765096 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:00.765112 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:00.765173 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:34:00.767344 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:00.767372 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:00.767942 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:00.768451 (+ 509us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:00.769183 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:00.770688 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:00.772719 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:00.773913 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:00.774202 (+ 289us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:00.775508 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:00.775523 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:00.775544 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:00.776289 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:00.776399 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:00.776436 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:00.776625 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:00.776912 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 23:34:00.777123 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":520,"threads_started":1}
I20260620 23:34:00.783643 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:00.786528 21016 raft_consensus.cc:348] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4293fae8c54a56b0a8330e896e3e2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260620 23:34:00.787027 21016 raft_consensus.cc:374] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:00.787246 21016 raft_consensus.cc:729] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da4293fae8c54a56b0a8330e896e3e2a, State: Initialized, Role: FOLLOWER
I20260620 23:34:00.788021 21016 consensus_queue.cc:260] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4293fae8c54a56b0a8330e896e3e2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260620 23:34:00.788368 21016 raft_consensus.cc:388] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:00.788520 21016 raft_consensus.cc:482] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:00.788722 21016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:00.791982 21016 raft_consensus.cc:504] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4293fae8c54a56b0a8330e896e3e2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260620 23:34:00.792557 21016 leader_election.cc:302] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da4293fae8c54a56b0a8330e896e3e2a; no voters:
I20260620 23:34:00.793354 21016 leader_election.cc:288] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:00.793550 21026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:00.794938 21026 raft_consensus.cc:686] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [term 1 LEADER]: Becoming Leader. State: Replica: da4293fae8c54a56b0a8330e896e3e2a, State: Running, Role: LEADER
I20260620 23:34:00.795945 21016 sys_catalog.cc:564] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:00.795866 21026 consensus_queue.cc:237] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4293fae8c54a56b0a8330e896e3e2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } }
I20260620 23:34:00.799278 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:00.789683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43731 (local address 127.0.0.1:40453)
0620 23:34:00.789983 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:00.789990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:00.790325 (+ 335us) server_negotiation.cc:407] Connection header received
0620 23:34:00.790534 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:00.790538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:00.790632 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:00.790755 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:00.791426 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:00.792365 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:00.794669 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:00.795500 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:00.795691 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:00.797470 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:00.797477 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:00.797482 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:00.797515 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:00.797612 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:00.797643 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:00.797784 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:00.798516 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 23:34:00.798608 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260620 23:34:00.801569 21027 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da4293fae8c54a56b0a8330e896e3e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4293fae8c54a56b0a8330e896e3e2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } } }
I20260620 23:34:00.802191 21027 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:00.803141 21037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:00.803030 21028 sys_catalog.cc:455] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader da4293fae8c54a56b0a8330e896e3e2a. Latest consensus state: current_term: 1 leader_uuid: "da4293fae8c54a56b0a8330e896e3e2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da4293fae8c54a56b0a8330e896e3e2a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40453 } } }
I20260620 23:34:00.803478 21028 sys_catalog.cc:458] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:00.814448 21037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:00.827313 21037 catalog_manager.cc:1347] Generated new cluster ID: 84fecf879cb74202a0e8c108faedc9f5
I20260620 23:34:00.827491 21037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:00.847460 21037 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:00.849298 21037 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:00.866237 21037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da4293fae8c54a56b0a8330e896e3e2a: Generated new TSK 0
I20260620 23:34:00.867430 21037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:00.876657 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20949
W20260620 23:34:00.878273 21011 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51275
W20260620 23:34:00.879457 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 20949: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:00.879658 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 20949
W20260620 23:34:00.898676 20934 connection.cc:535] client connection to 127.0.0.1:40453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40453): Connection reset by peer (error 104)
2026-06-20T23:34:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:00 dist-test-slave-v34r krb5kdc[21047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:00 dist-test-slave-v34r krb5kdc[21047](info): set up 2 sockets
Jun 20 23:34:00 dist-test-slave-v34r krb5kdc[21047](info): commencing operation
krb5kdc: starting...
W20260620 23:34:02.013693 387 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:02 dist-test-slave-v34r krb5kdc[21047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:02.320384 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:02.755257 21060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:02.755723 21060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:02.755909 21060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:02.764884 21060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:02.765197 21060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:02.765286 21060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:02.765362 21060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:02.765413 21060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:02.778218 21060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:02.780006 21060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:02.781751 21060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:02.797465 21068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:02.797600 21066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:02.797502 21065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:02.803184 21060 server_base.cc:1034] running on GCE node
I20260620 23:34:02.804379 21060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:02.806627 21060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:02.807920 21060 hybrid_clock.cc:648] HybridClock initialized: now 1781998442807827 us; error 70 us; skew 500 ppm
I20260620 23:34:02.808451 21060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:02 dist-test-slave-v34r krb5kdc[21047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:02.812677 21060 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:02.814936 21060 webserver.cc:460] Webserver started at http://127.0.0.1:36081/ using document root <none> and password file <none>
I20260620 23:34:02.815883 21060 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:02.816011 21060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:02.816372 21060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:02.820909 21060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "1bb047af89684bc89a368b3437cb369b"
format_stamp: "Formatted at 2026-06-20 23:34:02 on dist-test-slave-v34r"
server_key: "b1afd8804fd4447ba86219d03b0eda1a"
server_key_iv: "f60c9dd75f6e3671379dede0655ed101"
server_key_version: "encryptionkey@0"
I20260620 23:34:02.822105 21060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "1bb047af89684bc89a368b3437cb369b"
format_stamp: "Formatted at 2026-06-20 23:34:02 on dist-test-slave-v34r"
server_key: "b1afd8804fd4447ba86219d03b0eda1a"
server_key_iv: "f60c9dd75f6e3671379dede0655ed101"
server_key_version: "encryptionkey@0"
I20260620 23:34:02.829668 21060 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260620 23:34:02.834714 21075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:02.836737 21060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260620 23:34:02.837086 21060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "1bb047af89684bc89a368b3437cb369b"
format_stamp: "Formatted at 2026-06-20 23:34:02 on dist-test-slave-v34r"
server_key: "b1afd8804fd4447ba86219d03b0eda1a"
server_key_iv: "f60c9dd75f6e3671379dede0655ed101"
server_key_version: "encryptionkey@0"
I20260620 23:34:02.837407 21060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:02.844921 21060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:02.850395 21060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:02.850791 21060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:02.873206 21060 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32853
I20260620 23:34:02.873241 21126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32853 every 8 connection(s)
I20260620 23:34:02.875085 21060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:02.879225 21127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:02.880023 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21060
I20260620 23:34:02.880221 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:02.880839 387 external_mini_cluster.cc:1428] Setting key 9b85f2aa65fe6e51824833fa1124f030
I20260620 23:34:02.890617 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:02.883154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39963 (local address 127.0.0.1:32853)
0620 23:34:02.884229 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:02.884244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:02.884303 (+ 59us) server_negotiation.cc:407] Connection header received
0620 23:34:02.886368 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:02.886405 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:02.886928 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:02.887390 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:02.888551 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:02.888559 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:02.888576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:02.888591 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:02.888653 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:02.888679 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:02.888921 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:02.889152 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 23:34:02.889427 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":496,"threads_started":1}
I20260620 23:34:02.893620 21127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b: Bootstrap starting.
I20260620 23:34:02.895366 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:02.896862 21127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:02.897900 21127 log.cc:824] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:02.901108 21127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b: No bootstrap required, opened a new log
I20260620 23:34:02.903707 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:02.900540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37029 (local address 127.0.0.1:32853)
0620 23:34:02.900777 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:02.900784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:02.901286 (+ 502us) server_negotiation.cc:407] Connection header received
0620 23:34:02.901588 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:02.901593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:02.901661 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:02.901782 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:02.902852 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:02.902858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:02.902867 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:02.902895 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:02.902947 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:02.902965 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:02.903126 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:02.903244 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 23:34:02.903330 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 23:34:02.909430 21127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb047af89684bc89a368b3437cb369b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32853 } }
I20260620 23:34:02.909825 21127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:02.909938 21127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bb047af89684bc89a368b3437cb369b, State: Initialized, Role: FOLLOWER
I20260620 23:34:02.910578 21127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb047af89684bc89a368b3437cb369b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32853 } }
I20260620 23:34:02.910827 21127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:02.910984 21127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:02.911170 21127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:02.913892 21127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb047af89684bc89a368b3437cb369b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32853 } }
I20260620 23:34:02.914438 21127 leader_election.cc:302] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bb047af89684bc89a368b3437cb369b; no voters:
I20260620 23:34:02.915145 21127 leader_election.cc:288] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:02.915467 21137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:02.916422 21137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [term 1 LEADER]: Becoming Leader. State: Replica: 1bb047af89684bc89a368b3437cb369b, State: Running, Role: LEADER
I20260620 23:34:02.917178 21137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb047af89684bc89a368b3437cb369b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32853 } }
I20260620 23:34:02.917665 21127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:02.920951 21139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bb047af89684bc89a368b3437cb369b. Latest consensus state: current_term: 1 leader_uuid: "1bb047af89684bc89a368b3437cb369b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb047af89684bc89a368b3437cb369b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32853 } } }
I20260620 23:34:02.921545 21139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:02.921140 21138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bb047af89684bc89a368b3437cb369b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bb047af89684bc89a368b3437cb369b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32853 } } }
I20260620 23:34:02.921681 21138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:02.922703 21145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:02.929672 21145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:02.941381 21145 catalog_manager.cc:1347] Generated new cluster ID: a0205166bb9d435fb3f8f5fc4e491307
I20260620 23:34:02.941537 21145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:02.962845 21145 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:02.963585 21145 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:02.979331 21145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bb047af89684bc89a368b3437cb369b: Generated new TSK 0
I20260620 23:34:02.980878 21145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:02.988209 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21060
W20260620 23:34:02.991379 21122 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39963
W20260620 23:34:02.992591 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21060: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:02.992820 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21060
2026-06-20T23:34:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2117 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:03 dist-test-slave-v34r krb5kdc[21158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:03 dist-test-slave-v34r krb5kdc[21158](info): set up 2 sockets
Jun 20 23:34:03 dist-test-slave-v34r krb5kdc[21158](info): commencing operation
krb5kdc: starting...
W20260620 23:34:04.129482 387 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:04 dist-test-slave-v34r krb5kdc[21158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:04.435870 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:04.853331 21171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:04.853742 21171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:04.853848 21171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:04.863246 21171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:04.863411 21171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:04.863464 21171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:04.863523 21171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:04.863572 21171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:04.875180 21171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:04.877053 21171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:04.878760 21171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:04.894212 21179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:04.894290 21177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:04.894258 21176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:04.895118 21171 server_base.cc:1034] running on GCE node
I20260620 23:34:04.896266 21171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:04.898213 21171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:04.899479 21171 hybrid_clock.cc:648] HybridClock initialized: now 1781998444899433 us; error 48 us; skew 500 ppm
I20260620 23:34:04.899932 21171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:04 dist-test-slave-v34r krb5kdc[21158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:04.907783 21171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:04.910072 21171 webserver.cc:460] Webserver started at http://127.0.0.1:35821/ using document root <none> and password file <none>
I20260620 23:34:04.911060 21171 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:04.911245 21171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:04.911669 21171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:04.915673 21171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "7452a803c17842ffac919395d4f5170f"
format_stamp: "Formatted at 2026-06-20 23:34:04 on dist-test-slave-v34r"
server_key: "4ca054ac434fc272942cd235fa97e719"
server_key_iv: "c91d6f293397fc36509714338c0fe001"
server_key_version: "encryptionkey@0"
I20260620 23:34:04.916743 21171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "7452a803c17842ffac919395d4f5170f"
format_stamp: "Formatted at 2026-06-20 23:34:04 on dist-test-slave-v34r"
server_key: "4ca054ac434fc272942cd235fa97e719"
server_key_iv: "c91d6f293397fc36509714338c0fe001"
server_key_version: "encryptionkey@0"
I20260620 23:34:04.924124 21171 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260620 23:34:04.929607 21186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:04.931551 21171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:34:04.931847 21171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "7452a803c17842ffac919395d4f5170f"
format_stamp: "Formatted at 2026-06-20 23:34:04 on dist-test-slave-v34r"
server_key: "4ca054ac434fc272942cd235fa97e719"
server_key_iv: "c91d6f293397fc36509714338c0fe001"
server_key_version: "encryptionkey@0"
I20260620 23:34:04.932127 21171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:04.959573 21171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:04.965548 21171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:04.965943 21171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:04.990598 21171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35047
I20260620 23:34:04.990634 21237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35047 every 8 connection(s)
I20260620 23:34:04.992777 21171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:04.997226 21238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:04.999927 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21171
I20260620 23:34:05.000249 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:05.000756 387 external_mini_cluster.cc:1428] Setting key 668a7e866965e858be06f81fd0bdcd33
I20260620 23:34:05.011348 21238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f: Bootstrap starting.
I20260620 23:34:05.015503 21238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:05.016631 21238 log.cc:824] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f: Log is configured to *not* fsync() on all Append() calls
Jun 20 23:34:05 dist-test-slave-v34r krb5kdc[21158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:34:05.019915 21238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f: No bootstrap required, opened a new log
I20260620 23:34:05.028890 21238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7452a803c17842ffac919395d4f5170f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35047 } }
I20260620 23:34:05.029440 21238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:05.029585 21238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7452a803c17842ffac919395d4f5170f, State: Initialized, Role: FOLLOWER
I20260620 23:34:05.030333 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:05.003288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56679 (local address 127.0.0.1:35047)
0620 23:34:05.004353 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:05.004369 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:05.004426 (+ 57us) server_negotiation.cc:407] Connection header received
0620 23:34:05.006649 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:05.006693 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:05.007331 (+ 638us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:05.007803 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:05.009830 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:05.011715 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:05.013712 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:05.014596 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:05.014800 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:05.020052 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:05.020067 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:05.020099 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:05.020171 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:05.024314 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:05.025502 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:05.025513 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:05.025524 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:05.025794 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:05.026288 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:05.026292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:05.026295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:05.027096 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:05.027507 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:05.028036 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 23:34:05.028637 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":498,"threads_started":1}
I20260620 23:34:05.030238 21238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7452a803c17842ffac919395d4f5170f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35047 } }
I20260620 23:34:05.030544 21238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:05.030752 21238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:05.030951 21238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:05.033972 21238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7452a803c17842ffac919395d4f5170f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35047 } }
I20260620 23:34:05.034548 21238 leader_election.cc:302] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7452a803c17842ffac919395d4f5170f; no voters:
I20260620 23:34:05.035395 21238 leader_election.cc:288] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:05.035669 21243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:05.035972 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:05.037038 21243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [term 1 LEADER]: Becoming Leader. State: Replica: 7452a803c17842ffac919395d4f5170f, State: Running, Role: LEADER
I20260620 23:34:05.038062 21238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:05.038036 21243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7452a803c17842ffac919395d4f5170f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35047 } }
I20260620 23:34:05.042686 21244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7452a803c17842ffac919395d4f5170f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7452a803c17842ffac919395d4f5170f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35047 } } }
I20260620 23:34:05.043372 21244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:05.044145 21245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7452a803c17842ffac919395d4f5170f. Latest consensus state: current_term: 1 leader_uuid: "7452a803c17842ffac919395d4f5170f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7452a803c17842ffac919395d4f5170f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35047 } } }
I20260620 23:34:05.044600 21245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:05.051011 21253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:05.056881 21253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:05.084553 21253 catalog_manager.cc:1347] Generated new cluster ID: a1580b3f590d42ecb5bdf1c7f471798e
I20260620 23:34:05.084686 21253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:05.112044 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:05.089517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38441 (local address 127.0.0.1:35047)
0620 23:34:05.089866 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:05.089878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:05.093831 (+ 3953us) server_negotiation.cc:407] Connection header received
0620 23:34:05.094110 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:05.094115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:05.094206 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:05.094342 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:05.096188 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:05.097281 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:05.098826 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:05.099422 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:05.099620 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:05.102870 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:05.102877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:05.102881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:05.102961 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:05.107206 (+ 4245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:05.108230 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:05.108238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:05.108242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:05.108448 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:05.108922 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:05.108928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:05.108932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:05.109497 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:05.109792 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:05.111148 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0620 23:34:05.111466 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":121}
I20260620 23:34:05.112298 21253 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:05.114455 21253 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:05.130887 21253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7452a803c17842ffac919395d4f5170f: Generated new TSK 0
I20260620 23:34:05.132205 21253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:05.141594 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21171
I20260620 23:34:05.143672 21233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:05.255227 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21171
2026-06-20T23:34:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2261 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:05 dist-test-slave-v34r krb5kdc[21270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:05 dist-test-slave-v34r krb5kdc[21270](info): set up 2 sockets
Jun 20 23:34:05 dist-test-slave-v34r krb5kdc[21270](info): commencing operation
krb5kdc: starting...
W20260620 23:34:06.402148 387 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.003s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:06 dist-test-slave-v34r krb5kdc[21270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:06.709133 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:07.127584 21283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:07.128077 21283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:07.128283 21283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:07.136623 21283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:07.136870 21283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:07.136967 21283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:07.137123 21283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:07.137218 21283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:07.148772 21283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:07.150646 21283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:07.152426 21283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:07.167944 21289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:07.168022 21291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:07.167984 21288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:07.168854 21283 server_base.cc:1034] running on GCE node
I20260620 23:34:07.169945 21283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:07.171653 21283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:07.172927 21283 hybrid_clock.cc:648] HybridClock initialized: now 1781998447172852 us; error 83 us; skew 500 ppm
I20260620 23:34:07.173385 21283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:07 dist-test-slave-v34r krb5kdc[21270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:07.176810 21283 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:07.178591 21283 webserver.cc:460] Webserver started at http://127.0.0.1:39617/ using document root <none> and password file <none>
I20260620 23:34:07.179410 21283 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:07.179520 21283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:07.179795 21283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:07.182880 21283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "352130d1149740fc9ed26035dda2129c"
format_stamp: "Formatted at 2026-06-20 23:34:07 on dist-test-slave-v34r"
server_key: "7fb2520ff7849823bbfcbedc57266786"
server_key_iv: "34797c33de175cf9240509792b40a281"
server_key_version: "encryptionkey@0"
I20260620 23:34:07.183619 21283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "352130d1149740fc9ed26035dda2129c"
format_stamp: "Formatted at 2026-06-20 23:34:07 on dist-test-slave-v34r"
server_key: "7fb2520ff7849823bbfcbedc57266786"
server_key_iv: "34797c33de175cf9240509792b40a281"
server_key_version: "encryptionkey@0"
I20260620 23:34:07.189412 21283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:34:07.193452 21298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:07.195125 21283 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:34:07.195326 21283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "352130d1149740fc9ed26035dda2129c"
format_stamp: "Formatted at 2026-06-20 23:34:07 on dist-test-slave-v34r"
server_key: "7fb2520ff7849823bbfcbedc57266786"
server_key_iv: "34797c33de175cf9240509792b40a281"
server_key_version: "encryptionkey@0"
I20260620 23:34:07.195551 21283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:07.219110 21283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:07.224191 21283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:07.224568 21283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:07.248817 21283 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39941
I20260620 23:34:07.248843 21349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39941 every 8 connection(s)
I20260620 23:34:07.250604 21283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:07.256443 21350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:07.260673 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21283
I20260620 23:34:07.260882 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:07.261469 387 external_mini_cluster.cc:1428] Setting key 55987825ddaeb20991d694f67d0c4dac
I20260620 23:34:07.274278 21350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c: Bootstrap starting.
I20260620 23:34:07.279875 21350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:07.281373 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:07.263946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46165 (local address 127.0.0.1:39941)
0620 23:34:07.265096 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:07.265130 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:07.265205 (+ 75us) server_negotiation.cc:407] Connection header received
0620 23:34:07.267975 (+ 2770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:07.268020 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:07.268760 (+ 740us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:07.269494 (+ 734us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:07.270209 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:07.272499 (+ 2290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:07.274468 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:07.275669 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:07.275917 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:07.277366 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:07.277381 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:07.277408 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:07.278385 (+ 977us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:07.278522 (+ 137us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:07.278570 (+ 48us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:07.278755 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:07.279145 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 23:34:07.279544 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":463,"threads_started":1}
I20260620 23:34:07.281718 21350 log.cc:824] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:07.286510 21350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c: No bootstrap required, opened a new log
I20260620 23:34:07.287640 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:07.297355 21350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352130d1149740fc9ed26035dda2129c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260620 23:34:07.298043 21350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:07.298220 21350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352130d1149740fc9ed26035dda2129c, State: Initialized, Role: FOLLOWER
I20260620 23:34:07.299060 21350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352130d1149740fc9ed26035dda2129c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260620 23:34:07.299453 21350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:07.299655 21350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:07.299909 21350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:07.301227 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:07.293109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46995 (local address 127.0.0.1:39941)
0620 23:34:07.293380 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:07.293388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:07.293771 (+ 383us) server_negotiation.cc:407] Connection header received
0620 23:34:07.293966 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:07.293973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:07.294074 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:07.294251 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:07.294791 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:07.296014 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:07.297564 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:07.298344 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:07.298499 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:07.299867 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:07.299873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:07.299878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:07.299919 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:07.299990 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:07.300020 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:07.300146 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:07.300500 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 23:34:07.300651 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260620 23:34:07.303485 21350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352130d1149740fc9ed26035dda2129c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260620 23:34:07.304234 21350 leader_election.cc:302] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 352130d1149740fc9ed26035dda2129c; no voters:
I20260620 23:34:07.304958 21350 leader_election.cc:288] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:07.305260 21360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:07.306161 21360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [term 1 LEADER]: Becoming Leader. State: Replica: 352130d1149740fc9ed26035dda2129c, State: Running, Role: LEADER
I20260620 23:34:07.306962 21360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352130d1149740fc9ed26035dda2129c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } }
I20260620 23:34:07.308290 21350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:07.310273 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 352130d1149740fc9ed26035dda2129c. Latest consensus state: current_term: 1 leader_uuid: "352130d1149740fc9ed26035dda2129c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352130d1149740fc9ed26035dda2129c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } } }
I20260620 23:34:07.310640 21361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "352130d1149740fc9ed26035dda2129c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "352130d1149740fc9ed26035dda2129c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39941 } } }
I20260620 23:34:07.311686 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:07.311698 21361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:07.314719 21366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:07.323073 21366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:07.339895 21366 catalog_manager.cc:1347] Generated new cluster ID: 57dd61b8a5b640eb97616b5e4c54b090
I20260620 23:34:07.340324 21366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:07.356618 21366 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:07.359090 21366 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:07.372884 21366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 352130d1149740fc9ed26035dda2129c: Generated new TSK 0
I20260620 23:34:07.374058 21366 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:07.388381 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21283
W20260620 23:34:07.389778 21345 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46165
W20260620 23:34:07.391072 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21283: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:07.391335 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21283
2026-06-20T23:34:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2136 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:07 dist-test-slave-v34r krb5kdc[21381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:07 dist-test-slave-v34r krb5kdc[21381](info): set up 2 sockets
Jun 20 23:34:07 dist-test-slave-v34r krb5kdc[21381](info): commencing operation
krb5kdc: starting...
W20260620 23:34:08.545667 387 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:08 dist-test-slave-v34r krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:08.849155 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:09.227185 21394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:09.227576 21394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:09.227749 21394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:09.236070 21394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:09.236271 21394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:09.236397 21394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:09.236514 21394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:09.236603 21394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:09.248103 21394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:09.249778 21394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:09.251315 21394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:09.266563 21399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:09.266662 21400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:09.266669 21402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:09.267647 21394 server_base.cc:1034] running on GCE node
I20260620 23:34:09.268574 21394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:09.270243 21394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:09.271678 21394 hybrid_clock.cc:648] HybridClock initialized: now 1781998449271557 us; error 150 us; skew 500 ppm
I20260620 23:34:09.272210 21394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:09 dist-test-slave-v34r krb5kdc[21381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:09.276309 21394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:09.278442 21394 webserver.cc:460] Webserver started at http://127.0.0.1:35489/ using document root <none> and password file <none>
I20260620 23:34:09.279240 21394 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:09.279430 21394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:09.279780 21394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:09.283025 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "20ddf93502304da89299035e003991ff"
format_stamp: "Formatted at 2026-06-20 23:34:09 on dist-test-slave-v34r"
server_key: "45a3a2834e9177d72d744435f1a7c19d"
server_key_iv: "7467a2d08033be8cd196dee8eec06990"
server_key_version: "encryptionkey@0"
I20260620 23:34:09.283831 21394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "20ddf93502304da89299035e003991ff"
format_stamp: "Formatted at 2026-06-20 23:34:09 on dist-test-slave-v34r"
server_key: "45a3a2834e9177d72d744435f1a7c19d"
server_key_iv: "7467a2d08033be8cd196dee8eec06990"
server_key_version: "encryptionkey@0"
I20260620 23:34:09.289955 21394 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260620 23:34:09.294070 21409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:09.295635 21394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 23:34:09.295859 21394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "20ddf93502304da89299035e003991ff"
format_stamp: "Formatted at 2026-06-20 23:34:09 on dist-test-slave-v34r"
server_key: "45a3a2834e9177d72d744435f1a7c19d"
server_key_iv: "7467a2d08033be8cd196dee8eec06990"
server_key_version: "encryptionkey@0"
I20260620 23:34:09.296101 21394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:09.303181 21394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:09.308151 21394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:09.308485 21394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:09.331032 21394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42339
I20260620 23:34:09.331137 21460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42339 every 8 connection(s)
I20260620 23:34:09.332618 21394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:09.337716 21461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:09.343099 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21394
I20260620 23:34:09.343364 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:09.343973 387 external_mini_cluster.cc:1428] Setting key 6f8988a964bb5dfd075e6e1fdb8debb7
I20260620 23:34:09.354017 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:09.346715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38051 (local address 127.0.0.1:42339)
0620 23:34:09.347788 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:09.347804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:09.347855 (+ 51us) server_negotiation.cc:407] Connection header received
0620 23:34:09.349898 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:09.349926 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:09.350356 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:09.350846 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:09.352151 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:09.352161 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:09.352176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:09.352190 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:09.352247 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:09.352271 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:09.352551 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:09.352714 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 23:34:09.352914 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":804,"thread_start_us":485,"threads_started":1}
I20260620 23:34:09.355273 21461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff: Bootstrap starting.
I20260620 23:34:09.359108 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:09.360215 21461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:09.361979 21461 log.cc:824] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:09.366428 21461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff: No bootstrap required, opened a new log
I20260620 23:34:09.367727 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:09.364512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42605 (local address 127.0.0.1:42339)
0620 23:34:09.364722 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:09.364728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:09.365225 (+ 497us) server_negotiation.cc:407] Connection header received
0620 23:34:09.365423 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:09.365429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:09.365501 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:09.365693 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:09.366858 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:09.366863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:09.366867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:09.366887 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:09.366953 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:09.366976 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:09.367121 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:09.367369 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 23:34:09.367442 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 23:34:09.375589 21461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ddf93502304da89299035e003991ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 23:34:09.376307 21461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:09.376490 21461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20ddf93502304da89299035e003991ff, State: Initialized, Role: FOLLOWER
I20260620 23:34:09.377521 21461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ddf93502304da89299035e003991ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 23:34:09.377794 21461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:09.378007 21461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:09.378216 21461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:09.380661 21461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ddf93502304da89299035e003991ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 23:34:09.381259 21461 leader_election.cc:302] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20ddf93502304da89299035e003991ff; no voters:
I20260620 23:34:09.381982 21461 leader_election.cc:288] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:09.382316 21471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:09.383759 21471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [term 1 LEADER]: Becoming Leader. State: Replica: 20ddf93502304da89299035e003991ff, State: Running, Role: LEADER
I20260620 23:34:09.384541 21461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:09.384544 21471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ddf93502304da89299035e003991ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } }
I20260620 23:34:09.388656 21472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20ddf93502304da89299035e003991ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ddf93502304da89299035e003991ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } } }
I20260620 23:34:09.389269 21472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:09.389235 21473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20ddf93502304da89299035e003991ff. Latest consensus state: current_term: 1 leader_uuid: "20ddf93502304da89299035e003991ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20ddf93502304da89299035e003991ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42339 } } }
I20260620 23:34:09.389564 21473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:09.396503 21480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:09.402475 21480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:09.413689 21480 catalog_manager.cc:1347] Generated new cluster ID: 7dd9d4fe6b4a41ee972021b763a69e03
I20260620 23:34:09.413834 21480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:09.424552 21480 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:09.425295 21480 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:09.444537 21480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20ddf93502304da89299035e003991ff: Generated new TSK 0
I20260620 23:34:09.446208 21480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:09.456671 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21394
W20260620 23:34:09.458209 21456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38051
W20260620 23:34:09.459167 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21394: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:09.459317 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21394
2026-06-20T23:34:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2067 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:09 dist-test-slave-v34r krb5kdc[21492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:09 dist-test-slave-v34r krb5kdc[21492](info): set up 2 sockets
Jun 20 23:34:09 dist-test-slave-v34r krb5kdc[21492](info): commencing operation
krb5kdc: starting...
W20260620 23:34:10.617748 387 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.006s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:10 dist-test-slave-v34r krb5kdc[21492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:10.937021 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39241
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:11.369484 21505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:11.369935 21505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:11.370048 21505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:11.379642 21505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:11.379829 21505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:11.379899 21505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:11.379971 21505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:11.380020 21505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:11.391913 21505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:11.393687 21505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:11.395336 21505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:11.410583 21510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:11.410621 21511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:11.410558 21513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:11.412891 21505 server_base.cc:1034] running on GCE node
I20260620 23:34:11.414300 21505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:11.416343 21505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:11.417706 21505 hybrid_clock.cc:648] HybridClock initialized: now 1781998451417589 us; error 119 us; skew 500 ppm
I20260620 23:34:11.418215 21505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:11 dist-test-slave-v34r krb5kdc[21492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:11.422591 21505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:11.424865 21505 webserver.cc:460] Webserver started at http://127.0.0.1:33693/ using document root <none> and password file <none>
I20260620 23:34:11.425963 21505 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:11.426106 21505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:11.426441 21505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:11.430702 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "197998a84dc24b329506f2c6762d5c79"
format_stamp: "Formatted at 2026-06-20 23:34:11 on dist-test-slave-v34r"
server_key: "9fd3543d7475b6c5a135e526c369cf8b"
server_key_iv: "81538d30968a43bae5db1cc474120c1c"
server_key_version: "encryptionkey@0"
I20260620 23:34:11.431759 21505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "197998a84dc24b329506f2c6762d5c79"
format_stamp: "Formatted at 2026-06-20 23:34:11 on dist-test-slave-v34r"
server_key: "9fd3543d7475b6c5a135e526c369cf8b"
server_key_iv: "81538d30968a43bae5db1cc474120c1c"
server_key_version: "encryptionkey@0"
I20260620 23:34:11.439122 21505 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260620 23:34:11.444283 21520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:11.446502 21505 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:11.446832 21505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "197998a84dc24b329506f2c6762d5c79"
format_stamp: "Formatted at 2026-06-20 23:34:11 on dist-test-slave-v34r"
server_key: "9fd3543d7475b6c5a135e526c369cf8b"
server_key_iv: "81538d30968a43bae5db1cc474120c1c"
server_key_version: "encryptionkey@0"
I20260620 23:34:11.447111 21505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:11.472254 21505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:11.479151 21505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:11.479600 21505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:11.505760 21505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39241
I20260620 23:34:11.505812 21571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39241 every 8 connection(s)
I20260620 23:34:11.507619 21505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:11.512337 21572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:11.512701 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21505
I20260620 23:34:11.512912 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:11.513525 387 external_mini_cluster.cc:1428] Setting key b5f97e175e5f9cef8b1fcf0ce943e5a1
I20260620 23:34:11.529631 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79: Bootstrap starting.
Jun 20 23:34:11 dist-test-slave-v34r krb5kdc[21492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:34:11.534135 21572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:11.535328 21572 log.cc:824] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:11.538900 21572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79: No bootstrap required, opened a new log
I20260620 23:34:11.544739 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:11.516026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:54107 (local address 127.0.0.1:39241)
0620 23:34:11.517163 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:11.517182 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:11.517242 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:34:11.519910 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:11.519952 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:11.520626 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:11.521267 (+ 641us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:11.523403 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:11.525159 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:11.527438 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:11.528439 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:11.528649 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:11.534835 (+ 6186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:11.534848 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:11.534873 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:11.534942 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:11.538842 (+ 3900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:11.540050 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:11.540059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:11.540069 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:11.540320 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:11.540879 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:11.540883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:11.540886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:11.541767 (+ 881us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:11.542223 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:11.542812 (+ 589us) server_negotiation.cc:299] Negotiation successful
0620 23:34:11.543272 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":488,"threads_started":1}
I20260620 23:34:11.548466 21572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197998a84dc24b329506f2c6762d5c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39241 } }
I20260620 23:34:11.549081 21572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:11.549213 21572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197998a84dc24b329506f2c6762d5c79, State: Initialized, Role: FOLLOWER
I20260620 23:34:11.549950 21572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197998a84dc24b329506f2c6762d5c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39241 } }
I20260620 23:34:11.550199 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:11.550246 21572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:11.550383 21572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:11.550558 21572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:11.553565 21572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197998a84dc24b329506f2c6762d5c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39241 } }
I20260620 23:34:11.554432 21572 leader_election.cc:302] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 197998a84dc24b329506f2c6762d5c79; no voters:
I20260620 23:34:11.555302 21572 leader_election.cc:288] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:11.555439 21581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:11.559175 21572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:11.560668 21581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [term 1 LEADER]: Becoming Leader. State: Replica: 197998a84dc24b329506f2c6762d5c79, State: Running, Role: LEADER
I20260620 23:34:11.561784 21581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197998a84dc24b329506f2c6762d5c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39241 } }
I20260620 23:34:11.566128 21582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "197998a84dc24b329506f2c6762d5c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197998a84dc24b329506f2c6762d5c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39241 } } }
I20260620 23:34:11.566654 21582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:11.568029 21584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 197998a84dc24b329506f2c6762d5c79. Latest consensus state: current_term: 1 leader_uuid: "197998a84dc24b329506f2c6762d5c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197998a84dc24b329506f2c6762d5c79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39241 } } }
I20260620 23:34:11.568591 21584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:11.569130 21591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:11.582520 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:11.555736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:34045 (local address 127.0.0.1:39241)
0620 23:34:11.555988 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:11.556002 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:11.560141 (+ 4139us) server_negotiation.cc:407] Connection header received
0620 23:34:11.560360 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:11.560366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:11.560486 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:11.560610 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:11.562784 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:11.563690 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:11.567988 (+ 4298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:11.568609 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:11.568727 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:11.572662 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:11.572673 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:11.572681 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:11.572746 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:11.577580 (+ 4834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:11.579028 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:11.579034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:11.579037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:11.579207 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:11.580155 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:11.580160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:11.580164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:11.580681 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:11.581046 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:11.581620 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 23:34:11.581857 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":85}
I20260620 23:34:11.583240 21591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:11.592557 21591 catalog_manager.cc:1347] Generated new cluster ID: e348b5940e9d422fb7d171cecfaf7689
I20260620 23:34:11.592775 21591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:11.615912 21591 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:11.618372 21591 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:11.629197 21591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 197998a84dc24b329506f2c6762d5c79: Generated new TSK 0
I20260620 23:34:11.630635 21591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:11.639986 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21505
I20260620 23:34:11.641855 21567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:11.738216 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21505
W20260620 23:34:11.760403 21490 connection.cc:535] client connection to 127.0.0.1:39241 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39241): Connection reset by peer (error 104)
2026-06-20T23:34:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:11 dist-test-slave-v34r krb5kdc[21604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:11 dist-test-slave-v34r krb5kdc[21604](info): set up 2 sockets
Jun 20 23:34:11 dist-test-slave-v34r krb5kdc[21604](info): commencing operation
krb5kdc: starting...
W20260620 23:34:12.877624 387 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.003s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:13 dist-test-slave-v34r krb5kdc[21604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:13.182875 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:13.540834 21617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:13.541249 21617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:13.541350 21617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:13.548542 21617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:13.548668 21617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:13.548717 21617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:13.548771 21617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:13.548820 21617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:13.558782 21617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:13.560084 21617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:13.561375 21617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:13.574059 21622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:13.574239 21623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:13.574251 21625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:13.575553 21617 server_base.cc:1034] running on GCE node
I20260620 23:34:13.576673 21617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:13.578734 21617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:13.580063 21617 hybrid_clock.cc:648] HybridClock initialized: now 1781998453579991 us; error 75 us; skew 500 ppm
I20260620 23:34:13.580511 21617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:13 dist-test-slave-v34r krb5kdc[21604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:13.584303 21617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:13.586122 21617 webserver.cc:460] Webserver started at http://127.0.0.1:43437/ using document root <none> and password file <none>
I20260620 23:34:13.586844 21617 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:13.586989 21617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:13.587316 21617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:13.590453 21617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "f16675f8a61a4501ba011db200e11526"
format_stamp: "Formatted at 2026-06-20 23:34:13 on dist-test-slave-v34r"
server_key: "ae57604ae2cca5cdf8e37c3e894369ca"
server_key_iv: "df85b9446a744117c39a9094cf5e2259"
server_key_version: "encryptionkey@0"
I20260620 23:34:13.591228 21617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "f16675f8a61a4501ba011db200e11526"
format_stamp: "Formatted at 2026-06-20 23:34:13 on dist-test-slave-v34r"
server_key: "ae57604ae2cca5cdf8e37c3e894369ca"
server_key_iv: "df85b9446a744117c39a9094cf5e2259"
server_key_version: "encryptionkey@0"
I20260620 23:34:13.596920 21617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 23:34:13.600905 21632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:13.602802 21617 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:13.603021 21617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "f16675f8a61a4501ba011db200e11526"
format_stamp: "Formatted at 2026-06-20 23:34:13 on dist-test-slave-v34r"
server_key: "ae57604ae2cca5cdf8e37c3e894369ca"
server_key_iv: "df85b9446a744117c39a9094cf5e2259"
server_key_version: "encryptionkey@0"
I20260620 23:34:13.603228 21617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:13.615517 21617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:13.620036 21617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:13.620335 21617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:13.640872 21617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40799
I20260620 23:34:13.641139 21683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40799 every 8 connection(s)
I20260620 23:34:13.642406 21617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:13.644709 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21617
I20260620 23:34:13.644960 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:13.645606 387 external_mini_cluster.cc:1428] Setting key 847d4a60c8e68fe7d2c95614a36943e0
I20260620 23:34:13.648270 21684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:13.663700 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:13.647718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:34579 (local address 127.0.0.1:40799)
0620 23:34:13.648675 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:13.648694 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:13.648754 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:34:13.651447 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:13.651485 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:13.652186 (+ 701us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:13.652757 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:13.653434 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:13.655641 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:13.657131 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:13.658297 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:13.658521 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:13.659910 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:13.659922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:13.659959 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:13.660910 (+ 951us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:13.661082 (+ 172us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:13.661135 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:13.661339 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:13.661762 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 23:34:13.662029 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":356,"threads_started":1}
I20260620 23:34:13.665328 21684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526: Bootstrap starting.
I20260620 23:34:13.669330 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:13.670042 21684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:13.671649 21684 log.cc:824] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:13.676357 21684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526: No bootstrap required, opened a new log
I20260620 23:34:13.683965 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:13.675806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:56147 (local address 127.0.0.1:40799)
0620 23:34:13.676023 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:13.676031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:13.676544 (+ 513us) server_negotiation.cc:407] Connection header received
0620 23:34:13.676738 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:13.676744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:13.676864 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:13.677060 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:13.677670 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:13.678883 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:13.680438 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:13.681283 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:13.681459 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:13.682954 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:13.682961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:13.682967 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:13.682998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:13.683067 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:13.683095 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:13.683241 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:13.683437 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 23:34:13.683517 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 23:34:13.688419 21684 raft_consensus.cc:348] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16675f8a61a4501ba011db200e11526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40799 } }
I20260620 23:34:13.689100 21684 raft_consensus.cc:374] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:13.689216 21684 raft_consensus.cc:729] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16675f8a61a4501ba011db200e11526, State: Initialized, Role: FOLLOWER
I20260620 23:34:13.689965 21684 consensus_queue.cc:260] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16675f8a61a4501ba011db200e11526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40799 } }
I20260620 23:34:13.690232 21684 raft_consensus.cc:388] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:13.690357 21684 raft_consensus.cc:482] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:13.690517 21684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:13.693745 21684 raft_consensus.cc:504] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16675f8a61a4501ba011db200e11526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40799 } }
I20260620 23:34:13.694466 21684 leader_election.cc:302] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f16675f8a61a4501ba011db200e11526; no voters:
I20260620 23:34:13.695428 21684 leader_election.cc:288] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:13.695633 21694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:13.696713 21694 raft_consensus.cc:686] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [term 1 LEADER]: Becoming Leader. State: Replica: f16675f8a61a4501ba011db200e11526, State: Running, Role: LEADER
I20260620 23:34:13.697438 21694 consensus_queue.cc:237] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16675f8a61a4501ba011db200e11526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40799 } }
I20260620 23:34:13.698413 21684 sys_catalog.cc:564] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:13.701290 21696 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f16675f8a61a4501ba011db200e11526. Latest consensus state: current_term: 1 leader_uuid: "f16675f8a61a4501ba011db200e11526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16675f8a61a4501ba011db200e11526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40799 } } }
I20260620 23:34:13.701334 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f16675f8a61a4501ba011db200e11526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f16675f8a61a4501ba011db200e11526" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40799 } } }
I20260620 23:34:13.701777 21696 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:13.701860 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:13.705757 21701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:13.712960 21701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:13.724184 21701 catalog_manager.cc:1347] Generated new cluster ID: 00ca417fe0b04d8888a4a28a7197e95e
I20260620 23:34:13.724339 21701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:13.742842 21701 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:13.744546 21701 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:13.753764 21701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f16675f8a61a4501ba011db200e11526: Generated new TSK 0
I20260620 23:34:13.755156 21701 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:13.772156 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21617
W20260620 23:34:13.774041 21679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:34579
W20260620 23:34:13.774995 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21617: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:13.775166 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21617
W20260620 23:34:13.796660 21602 connection.cc:535] client connection to 127.0.0.1:40799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40799): Connection reset by peer (error 104)
2026-06-20T23:34:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:13 dist-test-slave-v34r krb5kdc[21715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:13 dist-test-slave-v34r krb5kdc[21715](info): set up 2 sockets
Jun 20 23:34:13 dist-test-slave-v34r krb5kdc[21715](info): commencing operation
krb5kdc: starting...
W20260620 23:34:14.913757 387 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:15 dist-test-slave-v34r krb5kdc[21715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:15.220609 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:15.564888 21728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:15.565351 21728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:15.565542 21728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:15.573235 21728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:15.573378 21728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:15.573465 21728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:15.573585 21728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:15.573653 21728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:15.584008 21728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:15.585378 21728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:15.586702 21728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:15.600107 21733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:15.600051 21736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:15.600181 21734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:15.603016 21728 server_base.cc:1034] running on GCE node
I20260620 23:34:15.603914 21728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:15.605567 21728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:15.606791 21728 hybrid_clock.cc:648] HybridClock initialized: now 1781998455606738 us; error 59 us; skew 500 ppm
I20260620 23:34:15.607195 21728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:15 dist-test-slave-v34r krb5kdc[21715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:15.610836 21728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:15.612715 21728 webserver.cc:460] Webserver started at http://127.0.0.1:44761/ using document root <none> and password file <none>
I20260620 23:34:15.613540 21728 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:15.613654 21728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:15.614003 21728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:15.617149 21728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "0f3f7a1c960544ff9aab87b003dd4694"
format_stamp: "Formatted at 2026-06-20 23:34:15 on dist-test-slave-v34r"
server_key: "8e609d41a10669caf3be5f296b326fe0"
server_key_iv: "60fd6a6c26f9c0bb25d6cb3c61fdb63e"
server_key_version: "encryptionkey@0"
I20260620 23:34:15.617916 21728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "0f3f7a1c960544ff9aab87b003dd4694"
format_stamp: "Formatted at 2026-06-20 23:34:15 on dist-test-slave-v34r"
server_key: "8e609d41a10669caf3be5f296b326fe0"
server_key_iv: "60fd6a6c26f9c0bb25d6cb3c61fdb63e"
server_key_version: "encryptionkey@0"
I20260620 23:34:15.623431 21728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 23:34:15.627234 21743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:15.628813 21728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 23:34:15.629048 21728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "0f3f7a1c960544ff9aab87b003dd4694"
format_stamp: "Formatted at 2026-06-20 23:34:15 on dist-test-slave-v34r"
server_key: "8e609d41a10669caf3be5f296b326fe0"
server_key_iv: "60fd6a6c26f9c0bb25d6cb3c61fdb63e"
server_key_version: "encryptionkey@0"
I20260620 23:34:15.629248 21728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:15.635334 21728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:15.639742 21728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:15.640065 21728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:15.661444 21728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46845
I20260620 23:34:15.661592 21794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46845 every 8 connection(s)
I20260620 23:34:15.662864 21728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:15.664454 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21728
I20260620 23:34:15.664667 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:15.665340 387 external_mini_cluster.cc:1428] Setting key a44ab76b8b2c43e0d9947503411845ca
I20260620 23:34:15.668628 21795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:15.674751 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:15.667598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:37459 (local address 127.0.0.1:46845)
0620 23:34:15.668522 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:15.668535 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:15.668584 (+ 49us) server_negotiation.cc:407] Connection header received
0620 23:34:15.670524 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:15.670552 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:15.670961 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:15.671365 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:15.672771 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:15.672778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:15.672797 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:15.672814 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:15.672911 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:15.672934 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:15.673285 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:15.673397 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 23:34:15.673695 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":424,"threads_started":1}
I20260620 23:34:15.678916 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:15.683518 21795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694: Bootstrap starting.
I20260620 23:34:15.686983 21798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:15.683871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:41767 (local address 127.0.0.1:46845)
0620 23:34:15.684090 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:15.684096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:15.684446 (+ 350us) server_negotiation.cc:407] Connection header received
0620 23:34:15.684691 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:15.684694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:15.684755 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:15.684843 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:15.686230 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:15.686234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:15.686237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:15.686253 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:15.686293 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:15.686307 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:15.686419 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:15.686587 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 23:34:15.686638 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:34:15.689033 21795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:15.690413 21795 log.cc:824] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:15.694506 21795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694: No bootstrap required, opened a new log
I20260620 23:34:15.704025 21795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3f7a1c960544ff9aab87b003dd4694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260620 23:34:15.704733 21795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:15.704859 21795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f3f7a1c960544ff9aab87b003dd4694, State: Initialized, Role: FOLLOWER
I20260620 23:34:15.705679 21795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3f7a1c960544ff9aab87b003dd4694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260620 23:34:15.705967 21795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:15.706140 21795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:15.706341 21795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:15.708923 21795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3f7a1c960544ff9aab87b003dd4694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260620 23:34:15.709702 21795 leader_election.cc:302] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f3f7a1c960544ff9aab87b003dd4694; no voters:
I20260620 23:34:15.710427 21795 leader_election.cc:288] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:15.710649 21805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:15.711535 21805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [term 1 LEADER]: Becoming Leader. State: Replica: 0f3f7a1c960544ff9aab87b003dd4694, State: Running, Role: LEADER
I20260620 23:34:15.712296 21805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3f7a1c960544ff9aab87b003dd4694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } }
I20260620 23:34:15.713352 21795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:15.716087 21806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f3f7a1c960544ff9aab87b003dd4694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3f7a1c960544ff9aab87b003dd4694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } } }
I20260620 23:34:15.716884 21806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:15.716240 21807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f3f7a1c960544ff9aab87b003dd4694. Latest consensus state: current_term: 1 leader_uuid: "0f3f7a1c960544ff9aab87b003dd4694" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f3f7a1c960544ff9aab87b003dd4694" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46845 } } }
I20260620 23:34:15.717278 21807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:15.718918 21814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:15.725883 21814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:15.737108 21814 catalog_manager.cc:1347] Generated new cluster ID: f4e1c078370940c5938a6276739b206d
I20260620 23:34:15.737298 21814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:15.755955 21814 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:15.756651 21814 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:15.766873 21814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f3f7a1c960544ff9aab87b003dd4694: Generated new TSK 0
I20260620 23:34:15.768113 21814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:15.783599 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21728
W20260620 23:34:15.785115 21790 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:37459
W20260620 23:34:15.786059 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21728: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:15.786254 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21728
2026-06-20T23:34:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:15 dist-test-slave-v34r krb5kdc[21826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:15 dist-test-slave-v34r krb5kdc[21826](info): set up 2 sockets
Jun 20 23:34:15 dist-test-slave-v34r krb5kdc[21826](info): commencing operation
krb5kdc: starting...
W20260620 23:34:16.917665 387 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:17 dist-test-slave-v34r krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:17.224079 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:17.566366 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:17.566952 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:17.567132 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:17.575770 21839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:17.575922 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:17.575976 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:17.576108 21839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:17.576180 21839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:17.587299 21839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:17.588699 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:17.590101 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:17.603185 21844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:17.603251 21847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:17.603271 21845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:17.604781 21839 server_base.cc:1034] running on GCE node
I20260620 23:34:17.605965 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:17.607662 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:17.609041 21839 hybrid_clock.cc:648] HybridClock initialized: now 1781998457608932 us; error 59 us; skew 500 ppm
I20260620 23:34:17.609452 21839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:17 dist-test-slave-v34r krb5kdc[21826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:17.622391 21839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:17.624370 21839 webserver.cc:460] Webserver started at http://127.0.0.1:40659/ using document root <none> and password file <none>
I20260620 23:34:17.625245 21839 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:17.625564 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:17.625944 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:17.629138 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "4fec4168857b4c9db67306a3b003651f"
format_stamp: "Formatted at 2026-06-20 23:34:17 on dist-test-slave-v34r"
server_key: "b55646a0c2644939e001fa5588d1d2cb"
server_key_iv: "ea95af127b1f5850b107264931e27a37"
server_key_version: "encryptionkey@0"
I20260620 23:34:17.629961 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "4fec4168857b4c9db67306a3b003651f"
format_stamp: "Formatted at 2026-06-20 23:34:17 on dist-test-slave-v34r"
server_key: "b55646a0c2644939e001fa5588d1d2cb"
server_key_iv: "ea95af127b1f5850b107264931e27a37"
server_key_version: "encryptionkey@0"
I20260620 23:34:17.636025 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 23:34:17.640132 21854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:17.641921 21839 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260620 23:34:17.642149 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "4fec4168857b4c9db67306a3b003651f"
format_stamp: "Formatted at 2026-06-20 23:34:17 on dist-test-slave-v34r"
server_key: "b55646a0c2644939e001fa5588d1d2cb"
server_key_iv: "ea95af127b1f5850b107264931e27a37"
server_key_version: "encryptionkey@0"
I20260620 23:34:17.642366 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:17.663342 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:17.668284 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:17.668624 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:17.689755 21839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37739
I20260620 23:34:17.689847 21905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37739 every 8 connection(s)
I20260620 23:34:17.691170 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:17.696246 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:17.696733 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21839
I20260620 23:34:17.696944 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:17.697424 387 external_mini_cluster.cc:1428] Setting key 9f7c6c8ae84e6313ca2bd07fa2fbf8e1
Jun 20 23:34:17 dist-test-slave-v34r krb5kdc[21826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 23:34:17.713104 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f: Bootstrap starting.
I20260620 23:34:17.717911 21906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:17.719632 21906 log.cc:824] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:17.723315 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:17.699485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:43085 (local address 127.0.0.1:37739)
0620 23:34:17.700620 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:17.700634 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:17.700679 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:34:17.702614 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:17.702640 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:17.703129 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:17.703584 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:17.705092 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:17.706482 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:17.707762 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:17.708511 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:17.708725 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:17.713806 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:17.713815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:17.713870 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:17.713919 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:17.717727 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:17.719046 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:17.719054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:17.719061 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:17.719267 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:17.719946 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:17.719953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:17.719955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:17.720851 (+ 896us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:17.721330 (+ 479us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:17.721737 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 23:34:17.722101 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":511,"threads_started":1}
I20260620 23:34:17.723930 21906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f: No bootstrap required, opened a new log
I20260620 23:34:17.728435 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:17.736167 21906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fec4168857b4c9db67306a3b003651f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 23:34:17.736747 21906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:17.736864 21906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fec4168857b4c9db67306a3b003651f, State: Initialized, Role: FOLLOWER
I20260620 23:34:17.737663 21906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fec4168857b4c9db67306a3b003651f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 23:34:17.738025 21906 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:17.738224 21906 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:17.738451 21906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:17.741801 21906 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fec4168857b4c9db67306a3b003651f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 23:34:17.742611 21906 leader_election.cc:302] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fec4168857b4c9db67306a3b003651f; no voters:
I20260620 23:34:17.743425 21906 leader_election.cc:288] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:17.743544 21916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:17.744747 21916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [term 1 LEADER]: Becoming Leader. State: Replica: 4fec4168857b4c9db67306a3b003651f, State: Running, Role: LEADER
I20260620 23:34:17.745600 21916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fec4168857b4c9db67306a3b003651f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } }
I20260620 23:34:17.746523 21906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:17.749424 21918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fec4168857b4c9db67306a3b003651f. Latest consensus state: current_term: 1 leader_uuid: "4fec4168857b4c9db67306a3b003651f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fec4168857b4c9db67306a3b003651f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } } }
I20260620 23:34:17.749898 21918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:17.749987 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fec4168857b4c9db67306a3b003651f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fec4168857b4c9db67306a3b003651f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37739 } } }
I20260620 23:34:17.750295 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:17.755218 21909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:17.734867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:56465 (local address 127.0.0.1:37739)
0620 23:34:17.735111 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:17.735118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:17.735518 (+ 400us) server_negotiation.cc:407] Connection header received
0620 23:34:17.735693 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:17.735700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:17.735808 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:17.735969 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:17.737916 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:17.739159 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:17.740515 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:17.741300 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:17.741471 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:17.744469 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:17.744487 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:17.744492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:17.744545 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:17.748121 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:17.749981 (+ 1860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:17.749992 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:17.749999 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:17.750307 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:17.751313 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:17.751322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:17.751329 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:17.751999 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:17.752459 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:17.754286 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0620 23:34:17.754615 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:34:17.765658 21924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:17.771099 21924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:17.782207 21924 catalog_manager.cc:1347] Generated new cluster ID: f09fe1a18c0948e4942d6a0265930bf1
I20260620 23:34:17.782424 21924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:17.792807 21924 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:17.794593 21924 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:17.803153 21924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fec4168857b4c9db67306a3b003651f: Generated new TSK 0
I20260620 23:34:17.804431 21924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:17.812306 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21839
I20260620 23:34:17.813930 21901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:17.907976 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21839
W20260620 23:34:17.926110 21824 connection.cc:535] client connection to 127.0.0.1:37739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37739): Connection reset by peer (error 104)
2026-06-20T23:34:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2125 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:18 dist-test-slave-v34r krb5kdc[21938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:18 dist-test-slave-v34r krb5kdc[21938](info): set up 2 sockets
Jun 20 23:34:18 dist-test-slave-v34r krb5kdc[21938](info): commencing operation
krb5kdc: starting...
W20260620 23:34:19.042291 387 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.007s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:19 dist-test-slave-v34r krb5kdc[21938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:19.336915 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:19.672642 21951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:19.673035 21951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:19.673213 21951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:19.681564 21951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:19.681756 21951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:19.681864 21951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:19.681988 21951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:19.682071 21951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:19.692672 21951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:19.694129 21951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:19.695534 21951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:19.708844 21959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:19.708935 21957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:19.708935 21956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:19.709794 21951 server_base.cc:1034] running on GCE node
I20260620 23:34:19.710675 21951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:19.712246 21951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:19.713461 21951 hybrid_clock.cc:648] HybridClock initialized: now 1781998459713402 us; error 60 us; skew 500 ppm
I20260620 23:34:19.713858 21951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:19 dist-test-slave-v34r krb5kdc[21938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:19.717273 21951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:19.718963 21951 webserver.cc:460] Webserver started at http://127.0.0.1:38403/ using document root <none> and password file <none>
I20260620 23:34:19.719694 21951 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:19.719802 21951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:19.720067 21951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:19.723238 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "b3e684e63bf64c6cb3fed26878d4e562"
format_stamp: "Formatted at 2026-06-20 23:34:19 on dist-test-slave-v34r"
server_key: "d5d48ce029c1d9eafba18c877e13f4f7"
server_key_iv: "ba170a0be169ac675ef9d3bf2ad14e28"
server_key_version: "encryptionkey@0"
I20260620 23:34:19.723973 21951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "b3e684e63bf64c6cb3fed26878d4e562"
format_stamp: "Formatted at 2026-06-20 23:34:19 on dist-test-slave-v34r"
server_key: "d5d48ce029c1d9eafba18c877e13f4f7"
server_key_iv: "ba170a0be169ac675ef9d3bf2ad14e28"
server_key_version: "encryptionkey@0"
I20260620 23:34:19.729785 21951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 23:34:19.733709 21966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:19.735316 21951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260620 23:34:19.735512 21951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "b3e684e63bf64c6cb3fed26878d4e562"
format_stamp: "Formatted at 2026-06-20 23:34:19 on dist-test-slave-v34r"
server_key: "d5d48ce029c1d9eafba18c877e13f4f7"
server_key_iv: "ba170a0be169ac675ef9d3bf2ad14e28"
server_key_version: "encryptionkey@0"
I20260620 23:34:19.735723 21951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:19.759685 21951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:19.763963 21951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:19.764261 21951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:19.785261 21951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40527
I20260620 23:34:19.785358 22017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40527 every 8 connection(s)
I20260620 23:34:19.786695 21951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:19.787035 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 21951
I20260620 23:34:19.787263 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:19.787920 387 external_mini_cluster.cc:1428] Setting key fffea6ca03ebf3c0d18ba6ad5439dedd
I20260620 23:34:19.791625 22018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:19.804600 22018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562: Bootstrap starting.
I20260620 23:34:19.805505 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:19.790454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:39893 (local address 127.0.0.1:40527)
0620 23:34:19.791411 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:19.791427 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:19.791483 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:34:19.793889 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:19.793917 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:19.794410 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:19.794902 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:19.795610 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:19.797723 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:19.799470 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:19.800402 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:19.800668 (+ 266us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:19.802359 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:19.802369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:19.802395 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:19.803129 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:19.803247 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:19.803283 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:19.803498 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:19.803851 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 23:34:19.804082 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":414,"threads_started":1}
I20260620 23:34:19.808277 22018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:19.809600 22018 log.cc:824] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:19.810789 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:19.812871 22018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562: No bootstrap required, opened a new log
I20260620 23:34:19.822389 22018 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e684e63bf64c6cb3fed26878d4e562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:34:19.822896 22018 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:19.823004 22018 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e684e63bf64c6cb3fed26878d4e562, State: Initialized, Role: FOLLOWER
I20260620 23:34:19.823652 22018 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e684e63bf64c6cb3fed26878d4e562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:34:19.823905 22018 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:19.824065 22018 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:19.824273 22018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:19.824657 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:19.816066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:43369 (local address 127.0.0.1:40527)
0620 23:34:19.816377 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:19.816384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:19.816873 (+ 489us) server_negotiation.cc:407] Connection header received
0620 23:34:19.817198 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:19.817204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:19.817313 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:19.817465 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:19.818171 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:19.819188 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:19.821258 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:19.821847 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:19.821981 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:19.823584 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:19.823590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:19.823594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:19.823621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:19.823710 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:19.823732 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:19.823895 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:19.824199 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 23:34:19.824292 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:34:19.827183 22018 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e684e63bf64c6cb3fed26878d4e562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:34:19.827741 22018 leader_election.cc:302] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3e684e63bf64c6cb3fed26878d4e562; no voters:
I20260620 23:34:19.828392 22018 leader_election.cc:288] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:19.828631 22028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:19.829641 22028 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [term 1 LEADER]: Becoming Leader. State: Replica: b3e684e63bf64c6cb3fed26878d4e562, State: Running, Role: LEADER
I20260620 23:34:19.830504 22028 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e684e63bf64c6cb3fed26878d4e562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } }
I20260620 23:34:19.830865 22018 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:19.836949 22029 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3e684e63bf64c6cb3fed26878d4e562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e684e63bf64c6cb3fed26878d4e562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } } }
I20260620 23:34:19.837047 22030 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3e684e63bf64c6cb3fed26878d4e562. Latest consensus state: current_term: 1 leader_uuid: "b3e684e63bf64c6cb3fed26878d4e562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3e684e63bf64c6cb3fed26878d4e562" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40527 } } }
I20260620 23:34:19.838486 22030 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:19.839727 22029 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:19.841786 22038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:19.847563 22038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:19.858469 22038 catalog_manager.cc:1347] Generated new cluster ID: fc7ef4750d724be8ab6e830e2e0a4e02
I20260620 23:34:19.858649 22038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:19.880940 22038 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:19.883443 22038 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:19.894840 22038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3e684e63bf64c6cb3fed26878d4e562: Generated new TSK 0
I20260620 23:34:19.896354 22038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:19.906531 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21951
W20260620 23:34:19.908528 22013 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:39893
W20260620 23:34:19.909416 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 21951: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:19.909621 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 21951
W20260620 23:34:19.928238 21936 connection.cc:535] client connection to 127.0.0.1:40527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40527): Connection reset by peer (error 104)
2026-06-20T23:34:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:20 dist-test-slave-v34r krb5kdc[22049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:20 dist-test-slave-v34r krb5kdc[22049](info): set up 2 sockets
Jun 20 23:34:20 dist-test-slave-v34r krb5kdc[22049](info): commencing operation
krb5kdc: starting...
W20260620 23:34:21.069585 387 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.010s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:21 dist-test-slave-v34r krb5kdc[22049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 23:34:21.368467 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37723
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:21.794334 22062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:21.794687 22062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:21.794838 22062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:21.803213 22062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:21.803339 22062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:21.803390 22062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:21.803483 22062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:21.803586 22062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:21.813970 22062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37723
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:21.815423 22062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:21.816825 22062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:21.830161 22070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:21.830047 22067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:21.830219 22068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:21.831013 22062 server_base.cc:1034] running on GCE node
I20260620 23:34:21.831868 22062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:21.833539 22062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:21.834822 22062 hybrid_clock.cc:648] HybridClock initialized: now 1781998461834772 us; error 60 us; skew 500 ppm
I20260620 23:34:21.835254 22062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:21 dist-test-slave-v34r krb5kdc[22049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:21.839145 22062 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 23:34:21.841184 22062 webserver.cc:460] Webserver started at http://127.0.0.1:32911/ using document root <none> and password file <none>
I20260620 23:34:21.842001 22062 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:21.842162 22062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:21.842496 22062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:21.845911 22062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "ccceeeff80f24b139b10521890a2e06b"
format_stamp: "Formatted at 2026-06-20 23:34:21 on dist-test-slave-v34r"
server_key: "3ea3fed3a456b31610ac81df9be577ea"
server_key_iv: "d2944b58261842489333f273ee6a42ff"
server_key_version: "encryptionkey@0"
I20260620 23:34:21.846724 22062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "ccceeeff80f24b139b10521890a2e06b"
format_stamp: "Formatted at 2026-06-20 23:34:21 on dist-test-slave-v34r"
server_key: "3ea3fed3a456b31610ac81df9be577ea"
server_key_iv: "d2944b58261842489333f273ee6a42ff"
server_key_version: "encryptionkey@0"
I20260620 23:34:21.852800 22062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:34:21.857131 22077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:21.858911 22062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:21.859163 22062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "ccceeeff80f24b139b10521890a2e06b"
format_stamp: "Formatted at 2026-06-20 23:34:21 on dist-test-slave-v34r"
server_key: "3ea3fed3a456b31610ac81df9be577ea"
server_key_iv: "d2944b58261842489333f273ee6a42ff"
server_key_version: "encryptionkey@0"
I20260620 23:34:21.859402 22062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:21.865916 22062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:21.870440 22062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:21.870755 22062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:21.893218 22062 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37723
I20260620 23:34:21.893302 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37723 every 8 connection(s)
I20260620 23:34:21.894776 22062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:21.899940 22129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:21.901085 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22062
I20260620 23:34:21.901288 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:21.901762 387 external_mini_cluster.cc:1428] Setting key 1489d4f98e7c993c3a86abf5b1cf5dc0
I20260620 23:34:21.911679 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:21.903848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:38157 (local address 127.0.0.1:37723)
0620 23:34:21.905070 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:21.905091 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:21.905158 (+ 67us) server_negotiation.cc:407] Connection header received
0620 23:34:21.907834 (+ 2676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:21.907871 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:21.908320 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:21.908797 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:21.909923 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:21.909929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:21.909945 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:21.909960 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:21.910022 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:21.910042 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:21.910300 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:21.910485 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 23:34:21.910666 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":464,"threads_started":1}
I20260620 23:34:21.916500 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:21.917358 22129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b: Bootstrap starting.
I20260620 23:34:21.921648 22129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:21.923494 22129 log.cc:824] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:21.924343 22132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:21.921627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.69:36375 (local address 127.0.0.1:37723)
0620 23:34:21.921873 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:21.921879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:21.922190 (+ 311us) server_negotiation.cc:407] Connection header received
0620 23:34:21.922407 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:21.922412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:21.922472 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:21.922587 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:21.923641 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:21.923655 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:21.923660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:21.923680 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:21.923726 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:21.923745 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:21.923880 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:21.923997 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 23:34:21.924060 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:34:21.927701 22129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b: No bootstrap required, opened a new log
I20260620 23:34:21.936859 22129 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccceeeff80f24b139b10521890a2e06b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260620 23:34:21.937633 22129 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:21.937793 22129 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccceeeff80f24b139b10521890a2e06b, State: Initialized, Role: FOLLOWER
I20260620 23:34:21.938761 22129 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccceeeff80f24b139b10521890a2e06b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260620 23:34:21.939154 22129 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:21.939342 22129 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:21.939590 22129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:21.942549 22129 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccceeeff80f24b139b10521890a2e06b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260620 23:34:21.943365 22129 leader_election.cc:302] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccceeeff80f24b139b10521890a2e06b; no voters:
I20260620 23:34:21.944180 22129 leader_election.cc:288] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:21.944406 22139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:21.945360 22139 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [term 1 LEADER]: Becoming Leader. State: Replica: ccceeeff80f24b139b10521890a2e06b, State: Running, Role: LEADER
I20260620 23:34:21.946058 22139 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccceeeff80f24b139b10521890a2e06b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } }
I20260620 23:34:21.946866 22129 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:21.949671 22141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccceeeff80f24b139b10521890a2e06b. Latest consensus state: current_term: 1 leader_uuid: "ccceeeff80f24b139b10521890a2e06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccceeeff80f24b139b10521890a2e06b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } } }
I20260620 23:34:21.950577 22140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccceeeff80f24b139b10521890a2e06b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccceeeff80f24b139b10521890a2e06b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37723 } } }
I20260620 23:34:21.951256 22140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:21.951265 22141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:21.954596 22147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:21.961963 22147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:21.974349 22147 catalog_manager.cc:1347] Generated new cluster ID: 9bf2e0c00c8442f49ed0c42a9078c869
I20260620 23:34:21.974495 22147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:22.001554 22147 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:22.002278 22147 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:22.011328 22147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccceeeff80f24b139b10521890a2e06b: Generated new TSK 0
I20260620 23:34:22.012368 22147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:22.026762 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22062
W20260620 23:34:22.029100 22124 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.69:38157
W20260620 23:34:22.030115 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22062: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:22.030282 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22062
2026-06-20T23:34:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:22 dist-test-slave-v34r krb5kdc[22160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:22 dist-test-slave-v34r krb5kdc[22160](info): set up 2 sockets
Jun 20 23:34:22 dist-test-slave-v34r krb5kdc[22160](info): commencing operation
krb5kdc: starting...
W20260620 23:34:23.157573 387 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:23 dist-test-slave-v34r krb5kdc[22160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:23.464231 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34401
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:40159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:23.878403 22173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:23.878722 22173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:23.878942 22173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:23.886485 22173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:23.886613 22173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:23.886701 22173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:23.886811 22173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:23.886898 22173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:23.897279 22173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:40159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:23.898638 22173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:23.900015 22173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:23.913805 22181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:23.913738 22179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:23.914450 22178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:23.915093 22173 server_base.cc:1034] running on GCE node
I20260620 23:34:23.916096 22173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:23.917819 22173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:23.919099 22173 hybrid_clock.cc:648] HybridClock initialized: now 1781998463919050 us; error 56 us; skew 500 ppm
I20260620 23:34:23.919574 22173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:23 dist-test-slave-v34r krb5kdc[22160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:23.926333 22173 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:23.928223 22173 webserver.cc:460] Webserver started at http://127.0.96.254:36663/ using document root <none> and password file <none>
I20260620 23:34:23.929138 22173 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:23.929301 22173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:23.929665 22173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:23.933152 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a"
format_stamp: "Formatted at 2026-06-20 23:34:23 on dist-test-slave-v34r"
server_key: "171ea86d3ef4627031a3cb6574242fb0"
server_key_iv: "fc72a26157e989e919f552a217404f51"
server_key_version: "encryptionkey@0"
I20260620 23:34:23.934054 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a"
format_stamp: "Formatted at 2026-06-20 23:34:23 on dist-test-slave-v34r"
server_key: "171ea86d3ef4627031a3cb6574242fb0"
server_key_iv: "fc72a26157e989e919f552a217404f51"
server_key_version: "encryptionkey@0"
I20260620 23:34:23.940308 22173 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:34:23.945134 22188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:23.947194 22173 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:23.947474 22173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a"
format_stamp: "Formatted at 2026-06-20 23:34:23 on dist-test-slave-v34r"
server_key: "171ea86d3ef4627031a3cb6574242fb0"
server_key_iv: "fc72a26157e989e919f552a217404f51"
server_key_version: "encryptionkey@0"
I20260620 23:34:23.947754 22173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:23.961261 22173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:23.965910 22173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:23.966226 22173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:23.988214 22173 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34401
I20260620 23:34:23.988314 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34401 every 8 connection(s)
I20260620 23:34:23.989835 22173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:23.994694 22240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:23.996469 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22173
I20260620 23:34:23.996713 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:23.997386 387 external_mini_cluster.cc:1428] Setting key 3d34824714de485a1b89e14f5e0e059a
I20260620 23:34:24.011257 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a: Bootstrap starting.
Jun 20 23:34:24 dist-test-slave-v34r krb5kdc[22160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:34:24.015849 22240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:24.017496 22240 log.cc:824] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:24.022197 22240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a: No bootstrap required, opened a new log
I20260620 23:34:24.025679 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:23.999904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49937 (local address 127.0.96.254:34401)
0620 23:34:24.000932 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:24.000951 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:24.001056 (+ 105us) server_negotiation.cc:407] Connection header received
0620 23:34:24.002996 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:24.003026 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:24.003502 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:24.003932 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:24.005535 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:24.006868 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:24.008253 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:24.009023 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:24.009235 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:24.014107 (+ 4872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:24.014115 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:24.014134 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:24.014177 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:24.018812 (+ 4635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:24.020352 (+ 1540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:24.020362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:24.020393 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:24.020629 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:24.021361 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:24.021367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:24.021370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:24.022124 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:24.022485 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:24.023609 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0620 23:34:24.024101 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":410,"threads_started":1}
I20260620 23:34:24.031774 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:24.033766 22240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34401 } }
I20260620 23:34:24.034242 22240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:24.034366 22240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26aeb3a99a7c4a94a3db6fd2f0b13c5a, State: Initialized, Role: FOLLOWER
I20260620 23:34:24.035104 22240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34401 } }
I20260620 23:34:24.035457 22240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:24.035643 22240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:24.035989 22240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:24.039407 22240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34401 } }
I20260620 23:34:24.039994 22240 leader_election.cc:302] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26aeb3a99a7c4a94a3db6fd2f0b13c5a; no voters:
I20260620 23:34:24.040707 22240 leader_election.cc:288] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:24.041105 22250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:24.042500 22250 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [term 1 LEADER]: Becoming Leader. State: Replica: 26aeb3a99a7c4a94a3db6fd2f0b13c5a, State: Running, Role: LEADER
I20260620 23:34:24.043397 22240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:24.043404 22250 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34401 } }
I20260620 23:34:24.053718 22251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34401 } } }
I20260620 23:34:24.054315 22251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:24.055642 22258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:24.057718 22252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26aeb3a99a7c4a94a3db6fd2f0b13c5a. Latest consensus state: current_term: 1 leader_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26aeb3a99a7c4a94a3db6fd2f0b13c5a" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34401 } } }
I20260620 23:34:24.058308 22252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:24.062213 22243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:24.038010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57189 (local address 127.0.96.254:34401)
0620 23:34:24.038237 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:24.038249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:24.038814 (+ 565us) server_negotiation.cc:407] Connection header received
0620 23:34:24.038982 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:24.038988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:24.039088 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:24.039233 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:24.041347 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:24.043045 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:24.045215 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:24.046470 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:24.046686 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:24.051761 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:24.051771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:24.051782 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:24.051865 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:24.056098 (+ 4233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:24.057504 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:24.057510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:24.057513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:24.057673 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:24.058372 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:24.058379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:24.058386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:24.059066 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:24.061047 (+ 1981us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:24.061308 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 23:34:24.061615 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:34:24.069906 22258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:24.082727 22258 catalog_manager.cc:1347] Generated new cluster ID: eb9b58ff9a854844a96a10c95eea8448
I20260620 23:34:24.082880 22258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:24.100514 22258 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:24.102070 22258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:24.116331 22258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26aeb3a99a7c4a94a3db6fd2f0b13c5a: Generated new TSK 0
I20260620 23:34:24.117465 22258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:24.137266 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22173
I20260620 23:34:24.138796 22235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:24.234915 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22173
W20260620 23:34:24.253414 22158 connection.cc:535] client connection to 127.0.96.254:34401 recv error: Network error: failed to read from TLS socket (remote: 127.0.96.254:34401): Connection reset by peer (error 104)
2026-06-20T23:34:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:24 dist-test-slave-v34r krb5kdc[22272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:24 dist-test-slave-v34r krb5kdc[22272](info): set up 2 sockets
Jun 20 23:34:24 dist-test-slave-v34r krb5kdc[22272](info): commencing operation
krb5kdc: starting...
W20260620 23:34:25.377950 387 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:25 dist-test-slave-v34r krb5kdc[22272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:25.684176 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:46173
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:43441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:26.091073 22285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:26.091440 22285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:26.091614 22285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:26.099974 22285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:26.100147 22285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:26.100237 22285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:26.100358 22285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:26.100449 22285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:26.111007 22285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:43441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:46173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:26.112500 22285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:26.113968 22285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:26.128002 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:26.128033 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:26.127985 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:26.129117 22285 server_base.cc:1034] running on GCE node
I20260620 23:34:26.129999 22285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:26.131644 22285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:26.132936 22285 hybrid_clock.cc:648] HybridClock initialized: now 1781998466132876 us; error 75 us; skew 500 ppm
I20260620 23:34:26.133440 22285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:26 dist-test-slave-v34r krb5kdc[22272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:26.136945 22285 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:26.138907 22285 webserver.cc:460] Webserver started at http://127.0.96.254:40361/ using document root <none> and password file <none>
I20260620 23:34:26.139714 22285 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:26.139871 22285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:26.140200 22285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:26.143391 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "197ba9f11a05420fba10537b1c41fc70"
format_stamp: "Formatted at 2026-06-20 23:34:26 on dist-test-slave-v34r"
server_key: "505799872fdb32ee2f7a8ff6128f5a1b"
server_key_iv: "f5f389860d13d7fb6016f0cfd5d9f64e"
server_key_version: "encryptionkey@0"
I20260620 23:34:26.144189 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "197ba9f11a05420fba10537b1c41fc70"
format_stamp: "Formatted at 2026-06-20 23:34:26 on dist-test-slave-v34r"
server_key: "505799872fdb32ee2f7a8ff6128f5a1b"
server_key_iv: "f5f389860d13d7fb6016f0cfd5d9f64e"
server_key_version: "encryptionkey@0"
I20260620 23:34:26.150125 22285 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260620 23:34:26.154640 22300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:26.156416 22285 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 23:34:26.156659 22285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "197ba9f11a05420fba10537b1c41fc70"
format_stamp: "Formatted at 2026-06-20 23:34:26 on dist-test-slave-v34r"
server_key: "505799872fdb32ee2f7a8ff6128f5a1b"
server_key_iv: "f5f389860d13d7fb6016f0cfd5d9f64e"
server_key_version: "encryptionkey@0"
I20260620 23:34:26.156908 22285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:26.196358 22285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:26.201120 22285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:26.201479 22285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:26.224488 22285 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:46173
I20260620 23:34:26.224588 22351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:46173 every 8 connection(s)
I20260620 23:34:26.225997 22285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:26.228067 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22285
I20260620 23:34:26.228291 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:26.229067 387 external_mini_cluster.cc:1428] Setting key 7a7db3ad05f118c40550a5dc38a57031
I20260620 23:34:26.232301 22352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:26.244270 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:26.231403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57085 (local address 127.0.96.254:46173)
0620 23:34:26.232445 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:26.232466 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:26.232527 (+ 61us) server_negotiation.cc:407] Connection header received
0620 23:34:26.234800 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:26.234825 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:26.235291 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:26.235756 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:26.236387 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:26.237878 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:26.239324 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:26.240130 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:26.240354 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:26.241781 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:26.241789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:26.241807 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:26.242414 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:26.242502 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:26.242528 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:26.242712 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:26.242985 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 23:34:26.243180 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":434,"threads_started":1}
I20260620 23:34:26.248817 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70: Bootstrap starting.
I20260620 23:34:26.248973 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:26.253322 22352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:26.255122 22352 log.cc:824] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:26.260118 22352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70: No bootstrap required, opened a new log
I20260620 23:34:26.261938 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:26.253748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59909 (local address 127.0.96.254:46173)
0620 23:34:26.254062 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:26.254070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:26.254500 (+ 430us) server_negotiation.cc:407] Connection header received
0620 23:34:26.254696 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:26.254706 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:26.254818 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:26.255009 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:26.255566 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:26.256675 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:26.258185 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:26.259011 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:26.259246 (+ 235us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:26.260729 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:26.260746 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:26.260752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:26.260788 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:26.260897 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:26.260937 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:26.261149 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:26.261417 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 23:34:26.261506 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260620 23:34:26.270123 22352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197ba9f11a05420fba10537b1c41fc70" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46173 } }
I20260620 23:34:26.270752 22352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:26.270889 22352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197ba9f11a05420fba10537b1c41fc70, State: Initialized, Role: FOLLOWER
I20260620 23:34:26.271932 22352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197ba9f11a05420fba10537b1c41fc70" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46173 } }
I20260620 23:34:26.272361 22352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:26.272570 22352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:26.272796 22352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:26.275614 22352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197ba9f11a05420fba10537b1c41fc70" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46173 } }
I20260620 23:34:26.276451 22352 leader_election.cc:302] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 197ba9f11a05420fba10537b1c41fc70; no voters:
I20260620 23:34:26.277384 22352 leader_election.cc:288] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:26.277544 22362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:26.278338 22362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [term 1 LEADER]: Becoming Leader. State: Replica: 197ba9f11a05420fba10537b1c41fc70, State: Running, Role: LEADER
I20260620 23:34:26.279222 22362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197ba9f11a05420fba10537b1c41fc70" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46173 } }
I20260620 23:34:26.280550 22352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:26.283087 22363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "197ba9f11a05420fba10537b1c41fc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197ba9f11a05420fba10537b1c41fc70" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46173 } } }
I20260620 23:34:26.283835 22363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:26.283522 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 197ba9f11a05420fba10537b1c41fc70. Latest consensus state: current_term: 1 leader_uuid: "197ba9f11a05420fba10537b1c41fc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "197ba9f11a05420fba10537b1c41fc70" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 46173 } } }
I20260620 23:34:26.284217 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:26.289480 22367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:26.295492 22367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:26.309196 22367 catalog_manager.cc:1347] Generated new cluster ID: 83b0deffa2df477e954fcef8d4ed1e53
I20260620 23:34:26.309363 22367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:26.357719 22367 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:26.359119 22367 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:26.373893 22367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 197ba9f11a05420fba10537b1c41fc70: Generated new TSK 0
I20260620 23:34:26.375409 22367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:26.394345 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22285
W20260620 23:34:26.396824 22347 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57085
W20260620 23:34:26.397928 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22285: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:26.398144 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22285
W20260620 23:34:26.419968 22270 connection.cc:535] client connection to 127.0.96.254:46173 recv error: Network error: failed to read from TLS socket (remote: 127.0.96.254:46173): Connection reset by peer (error 104)
2026-06-20T23:34:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2167 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:26 dist-test-slave-v34r krb5kdc[22383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:26 dist-test-slave-v34r krb5kdc[22383](info): set up 2 sockets
Jun 20 23:34:26 dist-test-slave-v34r krb5kdc[22383](info): commencing operation
krb5kdc: starting...
W20260620 23:34:27.561676 387 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.003s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:27 dist-test-slave-v34r krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:27.869069 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38879
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:34595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:28.331729 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:28.332089 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:28.332261 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:28.340492 22396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:28.340651 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:28.340755 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:28.340847 22396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:28.340932 22396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:28.351085 22396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:34595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:38879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:28.352394 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:28.353713 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:28.367280 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:28.367238 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:28.367511 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:28.368367 22396 server_base.cc:1034] running on GCE node
I20260620 23:34:28.369342 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:28.370982 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:28.372226 22396 hybrid_clock.cc:648] HybridClock initialized: now 1781998468372171 us; error 55 us; skew 500 ppm
I20260620 23:34:28.372659 22396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:28 dist-test-slave-v34r krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:28.376346 22396 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:28.378219 22396 webserver.cc:460] Webserver started at http://127.0.96.254:39875/ using document root <none> and password file <none>
I20260620 23:34:28.378979 22396 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:28.379125 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:28.379467 22396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:28.382761 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "4bf4a633c21d45a3af558e1ebdf35445"
format_stamp: "Formatted at 2026-06-20 23:34:28 on dist-test-slave-v34r"
server_key: "9b4e26de5aa49dbaa918e78fda66c790"
server_key_iv: "57140272c1c9d461847d94caf2a2767b"
server_key_version: "encryptionkey@0"
I20260620 23:34:28.383571 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "4bf4a633c21d45a3af558e1ebdf35445"
format_stamp: "Formatted at 2026-06-20 23:34:28 on dist-test-slave-v34r"
server_key: "9b4e26de5aa49dbaa918e78fda66c790"
server_key_iv: "57140272c1c9d461847d94caf2a2767b"
server_key_version: "encryptionkey@0"
I20260620 23:34:28.389473 22396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260620 23:34:28.393561 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:28.395102 22396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 23:34:28.395320 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "4bf4a633c21d45a3af558e1ebdf35445"
format_stamp: "Formatted at 2026-06-20 23:34:28 on dist-test-slave-v34r"
server_key: "9b4e26de5aa49dbaa918e78fda66c790"
server_key_iv: "57140272c1c9d461847d94caf2a2767b"
server_key_version: "encryptionkey@0"
I20260620 23:34:28.395506 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:28.401463 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:28.405529 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:28.405854 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:28.426721 22396 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:38879
I20260620 23:34:28.426761 22462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:38879 every 8 connection(s)
I20260620 23:34:28.428129 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:28.433193 22463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:28.436883 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22396
I20260620 23:34:28.437126 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:28.437630 387 external_mini_cluster.cc:1428] Setting key b1640cf4708eb7908332cda5f04cedba
I20260620 23:34:28.446516 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:28.439646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39117 (local address 127.0.96.254:38879)
0620 23:34:28.440586 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:28.440598 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:28.440643 (+ 45us) server_negotiation.cc:407] Connection header received
0620 23:34:28.442606 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:28.442631 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:28.443008 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:28.443386 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:28.444762 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:28.444770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:28.444782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:28.444799 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:28.444870 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:28.444892 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:28.445191 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:28.445305 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 23:34:28.445536 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":455,"threads_started":1}
I20260620 23:34:28.447875 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445: Bootstrap starting.
I20260620 23:34:28.451488 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:28.452211 22463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:28.453828 22463 log.cc:824] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:28.457451 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445: No bootstrap required, opened a new log
I20260620 23:34:28.461146 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:28.457861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35043 (local address 127.0.96.254:38879)
0620 23:34:28.458282 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:28.458292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:28.458580 (+ 288us) server_negotiation.cc:407] Connection header received
0620 23:34:28.458854 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:28.458860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:28.458944 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:28.459089 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:28.460301 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:28.460306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:28.460309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:28.460333 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:28.460385 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:28.460403 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:28.460577 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:28.460645 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 23:34:28.460735 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 23:34:28.466566 22463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf4a633c21d45a3af558e1ebdf35445" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38879 } }
I20260620 23:34:28.467065 22463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:28.467236 22463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf4a633c21d45a3af558e1ebdf35445, State: Initialized, Role: FOLLOWER
I20260620 23:34:28.467988 22463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf4a633c21d45a3af558e1ebdf35445" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38879 } }
I20260620 23:34:28.468266 22463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:28.468366 22463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:28.468495 22463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:28.470846 22463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf4a633c21d45a3af558e1ebdf35445" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38879 } }
I20260620 23:34:28.471378 22463 leader_election.cc:302] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf4a633c21d45a3af558e1ebdf35445; no voters:
I20260620 23:34:28.472213 22463 leader_election.cc:288] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:28.472472 22473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:28.473449 22473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [term 1 LEADER]: Becoming Leader. State: Replica: 4bf4a633c21d45a3af558e1ebdf35445, State: Running, Role: LEADER
I20260620 23:34:28.474256 22473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf4a633c21d45a3af558e1ebdf35445" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38879 } }
I20260620 23:34:28.474622 22463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:28.478547 22475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf4a633c21d45a3af558e1ebdf35445. Latest consensus state: current_term: 1 leader_uuid: "4bf4a633c21d45a3af558e1ebdf35445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf4a633c21d45a3af558e1ebdf35445" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38879 } } }
I20260620 23:34:28.479079 22475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:28.479653 22474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bf4a633c21d45a3af558e1ebdf35445" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf4a633c21d45a3af558e1ebdf35445" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 38879 } } }
I20260620 23:34:28.480163 22474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:28.480923 22482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:28.487795 22482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:28.499431 22482 catalog_manager.cc:1347] Generated new cluster ID: 89a77cfd1f9340c0a6db6fb39bdc628e
I20260620 23:34:28.499586 22482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:28.518767 22482 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:28.519368 22482 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:28.528286 22482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bf4a633c21d45a3af558e1ebdf35445: Generated new TSK 0
I20260620 23:34:28.529501 22482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:28.543155 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22396
W20260620 23:34:28.545300 22458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39117
W20260620 23:34:28.546229 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22396: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:28.546432 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22396
2026-06-20T23:34:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2144 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:28 dist-test-slave-v34r krb5kdc[22494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:28 dist-test-slave-v34r krb5kdc[22494](info): set up 2 sockets
Jun 20 23:34:28 dist-test-slave-v34r krb5kdc[22494](info): commencing operation
krb5kdc: starting...
W20260620 23:34:29.681620 387 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:29 dist-test-slave-v34r krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:29.989341 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35145
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:30.327297 22507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:30.327700 22507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:30.327865 22507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:30.335284 22507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:30.335409 22507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:30.335489 22507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:30.335589 22507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:30.335678 22507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:30.345737 22507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:30.347182 22507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:30.348528 22507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:30.362063 22513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:30.362138 22512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:30.362138 22515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:30.363651 22507 server_base.cc:1034] running on GCE node
I20260620 23:34:30.365191 22507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:30.367280 22507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:30.368611 22507 hybrid_clock.cc:648] HybridClock initialized: now 1781998470368540 us; error 59 us; skew 500 ppm
I20260620 23:34:30.369239 22507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:30 dist-test-slave-v34r krb5kdc[22494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:30.378804 22507 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:30.381122 22507 webserver.cc:460] Webserver started at http://127.0.96.254:33237/ using document root <none> and password file <none>
I20260620 23:34:30.382045 22507 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:30.382165 22507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:30.382493 22507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:30.386503 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "e02c56a237d141918ad7d977bbfe7358"
format_stamp: "Formatted at 2026-06-20 23:34:30 on dist-test-slave-v34r"
server_key: "881218765c238de9b6b643391192252a"
server_key_iv: "19f0e51c5807dd943af57da1b1300e4b"
server_key_version: "encryptionkey@0"
I20260620 23:34:30.387482 22507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "e02c56a237d141918ad7d977bbfe7358"
format_stamp: "Formatted at 2026-06-20 23:34:30 on dist-test-slave-v34r"
server_key: "881218765c238de9b6b643391192252a"
server_key_iv: "19f0e51c5807dd943af57da1b1300e4b"
server_key_version: "encryptionkey@0"
I20260620 23:34:30.394721 22507 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 23:34:30.399523 22522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:30.401512 22507 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:34:30.401747 22507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "e02c56a237d141918ad7d977bbfe7358"
format_stamp: "Formatted at 2026-06-20 23:34:30 on dist-test-slave-v34r"
server_key: "881218765c238de9b6b643391192252a"
server_key_iv: "19f0e51c5807dd943af57da1b1300e4b"
server_key_version: "encryptionkey@0"
I20260620 23:34:30.402004 22507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:30.427420 22507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:30.432941 22507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:30.433355 22507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:30.457486 22507 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:35145
I20260620 23:34:30.457508 22573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:35145 every 8 connection(s)
I20260620 23:34:30.459319 22507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:30.461247 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22507
I20260620 23:34:30.461460 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:30.461963 387 external_mini_cluster.cc:1428] Setting key a238325c7609a7c39c9c69133bb80f00
I20260620 23:34:30.464221 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:30.478852 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358: Bootstrap starting.
Jun 20 23:34:30 dist-test-slave-v34r krb5kdc[22494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:34:30.482682 22574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:30.484094 22574 log.cc:824] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:30.488351 22574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358: No bootstrap required, opened a new log
I20260620 23:34:30.490643 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:30.464071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36311 (local address 127.0.96.254:35145)
0620 23:34:30.465152 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:30.465168 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:30.465224 (+ 56us) server_negotiation.cc:407] Connection header received
0620 23:34:30.467257 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:30.467287 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:30.467867 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:30.468369 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:30.470393 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:30.472121 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:30.474180 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:30.475019 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:30.475194 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:30.480671 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:30.480683 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:30.480706 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:30.480767 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:30.485200 (+ 4433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:30.486408 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:30.486416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:30.486425 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:30.486696 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:30.487194 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:30.487199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:30.487203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:30.487931 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:30.488274 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:30.488818 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 23:34:30.489333 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":501,"threads_started":1}
I20260620 23:34:30.495793 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:30.497714 22574 raft_consensus.cc:348] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c56a237d141918ad7d977bbfe7358" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35145 } }
I20260620 23:34:30.498056 22574 raft_consensus.cc:374] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:30.498185 22574 raft_consensus.cc:729] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e02c56a237d141918ad7d977bbfe7358, State: Initialized, Role: FOLLOWER
I20260620 23:34:30.498821 22574 consensus_queue.cc:260] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c56a237d141918ad7d977bbfe7358" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35145 } }
I20260620 23:34:30.499059 22574 raft_consensus.cc:388] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:30.499226 22574 raft_consensus.cc:482] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:30.499445 22574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:30.502513 22574 raft_consensus.cc:504] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c56a237d141918ad7d977bbfe7358" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35145 } }
I20260620 23:34:30.503219 22574 leader_election.cc:302] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e02c56a237d141918ad7d977bbfe7358; no voters:
I20260620 23:34:30.503995 22574 leader_election.cc:288] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:30.504279 22584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:30.505515 22584 raft_consensus.cc:686] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [term 1 LEADER]: Becoming Leader. State: Replica: e02c56a237d141918ad7d977bbfe7358, State: Running, Role: LEADER
I20260620 23:34:30.506639 22574 sys_catalog.cc:564] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:30.506377 22584 consensus_queue.cc:237] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c56a237d141918ad7d977bbfe7358" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35145 } }
I20260620 23:34:30.510517 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e02c56a237d141918ad7d977bbfe7358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c56a237d141918ad7d977bbfe7358" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35145 } } }
I20260620 23:34:30.511032 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:30.511826 22586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e02c56a237d141918ad7d977bbfe7358. Latest consensus state: current_term: 1 leader_uuid: "e02c56a237d141918ad7d977bbfe7358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e02c56a237d141918ad7d977bbfe7358" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 35145 } } }
I20260620 23:34:30.512202 22586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:30.517124 22590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:30.522946 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:30.500803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55409 (local address 127.0.96.254:35145)
0620 23:34:30.501087 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:30.501096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:30.501686 (+ 590us) server_negotiation.cc:407] Connection header received
0620 23:34:30.501824 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:30.501829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:30.501929 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:30.502055 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:30.504182 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:30.505723 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:30.507708 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:30.508667 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:30.512676 (+ 4009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:30.513093 (+ 417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:30.513103 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:30.513112 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:30.513167 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:30.518308 (+ 5141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:30.519573 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:30.519581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:30.519586 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:30.519784 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:30.520182 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:30.520188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:30.520192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:30.520559 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:30.520869 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:30.522186 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0620 23:34:30.522476 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:34:30.526292 22590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:30.536932 22590 catalog_manager.cc:1347] Generated new cluster ID: 2ee3e5e3d0ea4332bc6945d4cb589c9c
I20260620 23:34:30.537120 22590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:30.552127 22590 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:30.553709 22590 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:30.563956 22590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e02c56a237d141918ad7d977bbfe7358: Generated new TSK 0
I20260620 23:34:30.565112 22590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:30.573696 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22507
I20260620 23:34:30.575839 22569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:30.669560 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22507
2026-06-20T23:34:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:30 dist-test-slave-v34r krb5kdc[22606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:30 dist-test-slave-v34r krb5kdc[22606](info): set up 2 sockets
Jun 20 23:34:30 dist-test-slave-v34r krb5kdc[22606](info): commencing operation
krb5kdc: starting...
W20260620 23:34:31.809689 387 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.001s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:32 dist-test-slave-v34r krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:32.125391 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36617
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:41661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:36617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:32.546868 22619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:32.547309 22619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:32.547490 22619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:32.555733 22619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:32.555943 22619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:32.556067 22619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:32.556187 22619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:32.556270 22619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:32.568010 22619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:41661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:36617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:36617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:32.569830 22619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:32.571535 22619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:32.587538 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:32.588120 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:32.588174 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:32.589119 22619 server_base.cc:1034] running on GCE node
I20260620 23:34:32.590037 22619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:32.591724 22619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:32.593101 22619 hybrid_clock.cc:648] HybridClock initialized: now 1781998472592958 us; error 69 us; skew 500 ppm
I20260620 23:34:32.593569 22619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:32 dist-test-slave-v34r krb5kdc[22606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:32.597936 22619 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:32.600603 22619 webserver.cc:460] Webserver started at http://127.0.96.254:37257/ using document root <none> and password file <none>
I20260620 23:34:32.601504 22619 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:32.601634 22619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:32.601938 22619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:32.605342 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "5b9f70d8c1a34ed59196101fea06ace2"
format_stamp: "Formatted at 2026-06-20 23:34:32 on dist-test-slave-v34r"
server_key: "82fb7a00caa3a07462dc75111b672237"
server_key_iv: "b9563658df802028e0188debf9ce5417"
server_key_version: "encryptionkey@0"
I20260620 23:34:32.606159 22619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "5b9f70d8c1a34ed59196101fea06ace2"
format_stamp: "Formatted at 2026-06-20 23:34:32 on dist-test-slave-v34r"
server_key: "82fb7a00caa3a07462dc75111b672237"
server_key_iv: "b9563658df802028e0188debf9ce5417"
server_key_version: "encryptionkey@0"
I20260620 23:34:32.612723 22619 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:34:32.617131 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:32.618844 22619 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:32.619059 22619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "5b9f70d8c1a34ed59196101fea06ace2"
format_stamp: "Formatted at 2026-06-20 23:34:32 on dist-test-slave-v34r"
server_key: "82fb7a00caa3a07462dc75111b672237"
server_key_iv: "b9563658df802028e0188debf9ce5417"
server_key_version: "encryptionkey@0"
I20260620 23:34:32.619318 22619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:32.633579 22619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:32.638159 22619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:32.638484 22619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:32.659770 22619 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:36617
I20260620 23:34:32.659806 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:36617 every 8 connection(s)
I20260620 23:34:32.661352 22619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:32.666013 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:32.669428 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22619
I20260620 23:34:32.669628 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:32.670120 387 external_mini_cluster.cc:1428] Setting key a8d1502ae0898a5e48f65f3b314d081d
I20260620 23:34:32.682245 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2: Bootstrap starting.
I20260620 23:34:32.687126 22686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:32.688220 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:32.672312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48655 (local address 127.0.96.254:36617)
0620 23:34:32.673351 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:32.673374 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:32.673447 (+ 73us) server_negotiation.cc:407] Connection header received
0620 23:34:32.676070 (+ 2623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:32.676109 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:32.676787 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:32.677407 (+ 620us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:32.678082 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:32.680299 (+ 2217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:32.681946 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:32.683097 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:32.683331 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:32.684620 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:32.684634 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:32.684658 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:32.685562 (+ 904us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:32.685695 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:32.685735 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:32.685894 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:32.686144 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 23:34:32.686493 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":387,"threads_started":1}
I20260620 23:34:32.688781 22686 log.cc:824] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:32.692564 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2: No bootstrap required, opened a new log
I20260620 23:34:32.692714 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:32.704968 22686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9f70d8c1a34ed59196101fea06ace2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36617 } }
I20260620 23:34:32.705731 22686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:32.705868 22686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b9f70d8c1a34ed59196101fea06ace2, State: Initialized, Role: FOLLOWER
I20260620 23:34:32.706710 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:32.698571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41027 (local address 127.0.96.254:36617)
0620 23:34:32.698854 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:32.698864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:32.699414 (+ 550us) server_negotiation.cc:407] Connection header received
0620 23:34:32.699645 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:32.699651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:32.699740 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:32.699831 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:32.700518 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:32.701349 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:32.703334 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:32.704150 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:32.704275 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:32.705853 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:32.705863 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:32.705868 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:32.705897 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:32.705942 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:32.705978 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:32.706080 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:32.706266 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 23:34:32.706365 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 23:34:32.706841 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9f70d8c1a34ed59196101fea06ace2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36617 } }
I20260620 23:34:32.707182 22686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:32.707356 22686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:32.707583 22686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:32.710481 22686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9f70d8c1a34ed59196101fea06ace2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36617 } }
I20260620 23:34:32.711166 22686 leader_election.cc:302] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b9f70d8c1a34ed59196101fea06ace2; no voters:
I20260620 23:34:32.711913 22686 leader_election.cc:288] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:32.712067 22696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:32.712889 22696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b9f70d8c1a34ed59196101fea06ace2, State: Running, Role: LEADER
I20260620 23:34:32.713637 22696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9f70d8c1a34ed59196101fea06ace2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36617 } }
I20260620 23:34:32.714648 22686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:32.716866 22698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b9f70d8c1a34ed59196101fea06ace2. Latest consensus state: current_term: 1 leader_uuid: "5b9f70d8c1a34ed59196101fea06ace2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9f70d8c1a34ed59196101fea06ace2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36617 } } }
I20260620 23:34:32.716854 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b9f70d8c1a34ed59196101fea06ace2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b9f70d8c1a34ed59196101fea06ace2" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 36617 } } }
I20260620 23:34:32.717432 22698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:32.717602 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:32.718415 22702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:32.723973 22702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:32.734083 22702 catalog_manager.cc:1347] Generated new cluster ID: 0b1b9c9cf68d4f22bd37bea9ecb370bd
I20260620 23:34:32.734297 22702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:32.763448 22702 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:32.765611 22702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:32.773907 22702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b9f70d8c1a34ed59196101fea06ace2: Generated new TSK 0
I20260620 23:34:32.775043 22702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:32.793864 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22619
W20260620 23:34:32.795946 22681 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48655
W20260620 23:34:32.796851 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22619: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:32.797048 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22619
2026-06-20T23:34:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2127 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:32 dist-test-slave-v34r krb5kdc[22717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:32 dist-test-slave-v34r krb5kdc[22717](info): set up 2 sockets
Jun 20 23:34:32 dist-test-slave-v34r krb5kdc[22717](info): commencing operation
krb5kdc: starting...
W20260620 23:34:33.933733 387 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:34 dist-test-slave-v34r krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:34.240118 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34309
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:46119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:34309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:34.656607 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:34.657115 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:34.657315 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:34.665653 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:34.665880 22730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:34.666018 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:34.666139 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:34.666231 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:34.677695 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:46119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:34309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:34.679605 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:34.681421 22730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:34.695784 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:34.695839 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:34.695851 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:34.697356 22730 server_base.cc:1034] running on GCE node
I20260620 23:34:34.698720 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:34.700765 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:34.702047 22730 hybrid_clock.cc:648] HybridClock initialized: now 1781998474701983 us; error 63 us; skew 500 ppm
I20260620 23:34:34.702508 22730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:34 dist-test-slave-v34r krb5kdc[22717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998474, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:34.706681 22730 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:34.708802 22730 webserver.cc:460] Webserver started at http://127.0.96.254:38019/ using document root <none> and password file <none>
I20260620 23:34:34.709824 22730 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:34.709992 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:34.710422 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:34.714340 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "4730a87736d24cc18216adfb6e97378c"
format_stamp: "Formatted at 2026-06-20 23:34:34 on dist-test-slave-v34r"
server_key: "339a331e555e22107effacea524e0fda"
server_key_iv: "63ab732429cf1b0b57b4bb84967fcd96"
server_key_version: "encryptionkey@0"
I20260620 23:34:34.715401 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "4730a87736d24cc18216adfb6e97378c"
format_stamp: "Formatted at 2026-06-20 23:34:34 on dist-test-slave-v34r"
server_key: "339a331e555e22107effacea524e0fda"
server_key_iv: "63ab732429cf1b0b57b4bb84967fcd96"
server_key_version: "encryptionkey@0"
I20260620 23:34:34.722621 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260620 23:34:34.727897 22745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:34.730171 22730 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260620 23:34:34.730443 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "4730a87736d24cc18216adfb6e97378c"
format_stamp: "Formatted at 2026-06-20 23:34:34 on dist-test-slave-v34r"
server_key: "339a331e555e22107effacea524e0fda"
server_key_iv: "63ab732429cf1b0b57b4bb84967fcd96"
server_key_version: "encryptionkey@0"
I20260620 23:34:34.730705 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:34.737974 22730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:34.743171 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:34.743531 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:34.766363 22730 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:34309
I20260620 23:34:34.766415 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:34309 every 8 connection(s)
I20260620 23:34:34.768265 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:34.772192 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22730
I20260620 23:34:34.772403 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:34.772974 387 external_mini_cluster.cc:1428] Setting key 19b019347f74083a54d586c0786425f0
I20260620 23:34:34.773231 22797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:34.782766 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:34.775246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54433 (local address 127.0.96.254:34309)
0620 23:34:34.776253 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:34.776268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:34.776328 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:34:34.778469 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:34.778505 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:34.778974 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:34.779404 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:34.780725 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:34.780733 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:34.780748 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:34.780768 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:34.780839 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:34.780869 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:34.781176 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:34.781335 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 23:34:34.781614 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":444,"threads_started":1}
I20260620 23:34:34.787360 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:34.788501 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c: Bootstrap starting.
I20260620 23:34:34.792862 22797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:34.794387 22797 log.cc:824] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:34.797072 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:34.793812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46863 (local address 127.0.96.254:34309)
0620 23:34:34.794111 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:34.794118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:34.794569 (+ 451us) server_negotiation.cc:407] Connection header received
0620 23:34:34.794776 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:34.794781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:34.794851 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:34.794985 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:34.796185 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:34.796190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:34.796194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 23:34:34.796211 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:34.796259 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 23:34:34.796276 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:34.796383 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:34.796553 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 23:34:34.796616 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 23:34:34.798290 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c: No bootstrap required, opened a new log
I20260620 23:34:34.806442 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730a87736d24cc18216adfb6e97378c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34309 } }
I20260620 23:34:34.806870 22797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 23:34:34.806984 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4730a87736d24cc18216adfb6e97378c, State: Initialized, Role: FOLLOWER
I20260620 23:34:34.807608 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730a87736d24cc18216adfb6e97378c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34309 } }
I20260620 23:34:34.807878 22797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 23:34:34.808036 22797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 23:34:34.808243 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:34.810693 22797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730a87736d24cc18216adfb6e97378c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34309 } }
I20260620 23:34:34.811251 22797 leader_election.cc:302] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4730a87736d24cc18216adfb6e97378c; no voters:
I20260620 23:34:34.811988 22797 leader_election.cc:288] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 23:34:34.812222 22807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:34.814004 22807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [term 1 LEADER]: Becoming Leader. State: Replica: 4730a87736d24cc18216adfb6e97378c, State: Running, Role: LEADER
I20260620 23:34:34.814543 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:34.814905 22807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730a87736d24cc18216adfb6e97378c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34309 } }
I20260620 23:34:34.819170 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4730a87736d24cc18216adfb6e97378c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730a87736d24cc18216adfb6e97378c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34309 } } }
I20260620 23:34:34.819331 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4730a87736d24cc18216adfb6e97378c. Latest consensus state: current_term: 1 leader_uuid: "4730a87736d24cc18216adfb6e97378c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4730a87736d24cc18216adfb6e97378c" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 34309 } } }
I20260620 23:34:34.819855 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:34.819707 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:34.822966 22816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:34.829023 22816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:34.841768 22816 catalog_manager.cc:1347] Generated new cluster ID: b24975ccb60c4903a4f80b60983c48df
I20260620 23:34:34.841928 22816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:34.885830 22816 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:34.886404 22816 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:34.901292 22816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4730a87736d24cc18216adfb6e97378c: Generated new TSK 0
I20260620 23:34:34.902206 22816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 23:34:34.910977 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22730
W20260620 23:34:34.912521 22792 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54433
W20260620 23:34:34.913270 387 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22730: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 23:34:34.913488 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22730
2026-06-20T23:34:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2119 ms)
[----------] 18 tests from AuthTokenIssuingTest (38240 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:35 dist-test-slave-v34r krb5kdc[22828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:35 dist-test-slave-v34r krb5kdc[22828](info): set up 2 sockets
Jun 20 23:34:35 dist-test-slave-v34r krb5kdc[22828](info): commencing operation
krb5kdc: starting...
W20260620 23:34:36.053977 387 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:36 dist-test-slave-v34r krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:36.377699 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42019
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42019,127.0.96.253:37983,127.0.96.252:36353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:36.806567 22841 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:34:36.807011 22841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:36.807083 22841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:36.807180 22841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:36.815514 22841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:36.815641 22841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:36.815704 22841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:36.815754 22841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:36.815826 22841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:36.827474 22841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42019,127.0.96.253:37983,127.0.96.252:36353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:42019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:36.829270 22841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:36.830926 22841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:36.845247 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:36.845396 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:36.845613 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:36.847065 22841 server_base.cc:1034] running on GCE node
I20260620 23:34:36.848035 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:36.849892 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:36.851148 22841 hybrid_clock.cc:648] HybridClock initialized: now 1781998476851057 us; error 69 us; skew 500 ppm
I20260620 23:34:36.851585 22841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:36 dist-test-slave-v34r krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:36.858803 22841 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:36.860901 22841 webserver.cc:460] Webserver started at http://127.0.96.254:44627/ using document root <none> and password file <none>
I20260620 23:34:36.861892 22841 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:36.862066 22841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:36.862458 22841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:36.866207 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "519d97225ca04828af80fe98cfc325af"
format_stamp: "Formatted at 2026-06-20 23:34:36 on dist-test-slave-v34r"
server_key: "fc085f1e1050ab1038c474c47e589994"
server_key_iv: "cdbb726d693369b610305ce6d650c8a3"
server_key_version: "encryptionkey@0"
I20260620 23:34:36.867172 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "519d97225ca04828af80fe98cfc325af"
format_stamp: "Formatted at 2026-06-20 23:34:36 on dist-test-slave-v34r"
server_key: "fc085f1e1050ab1038c474c47e589994"
server_key_iv: "cdbb726d693369b610305ce6d650c8a3"
server_key_version: "encryptionkey@0"
I20260620 23:34:36.873899 22841 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 23:34:36.878712 22856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:36.880590 22841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:34:36.880880 22841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "519d97225ca04828af80fe98cfc325af"
format_stamp: "Formatted at 2026-06-20 23:34:36 on dist-test-slave-v34r"
server_key: "fc085f1e1050ab1038c474c47e589994"
server_key_iv: "cdbb726d693369b610305ce6d650c8a3"
server_key_version: "encryptionkey@0"
I20260620 23:34:36.881281 22841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:36.903322 22841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:36.908633 22841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:36.909107 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:36.933670 22841 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:42019
I20260620 23:34:36.933686 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:42019 every 8 connection(s)
I20260620 23:34:36.935549 22841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:36.937240 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22841
I20260620 23:34:36.937475 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:36.938145 387 external_mini_cluster.cc:1428] Setting key d62275343a7a813a12ee5eee5472b3be
I20260620 23:34:36.939891 22908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:36.946923 22908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:36 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.253@KRBTEST.COM" created.
I20260620 23:34:36.977981 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:36.950863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.0.96.254:42019)
0620 23:34:36.953850 (+ 2987us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:36.953866 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:36.953951 (+ 85us) server_negotiation.cc:407] Connection header received
0620 23:34:36.955698 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:36.955712 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:36.956179 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:36.956482 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:36.958048 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:36.960146 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:36.961939 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:36.963088 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:36.963345 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:36.967708 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:36.967727 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:36.967755 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:36.967824 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:36.972196 (+ 4372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:36.973210 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:36.973220 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:36.973223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:36.973433 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:36.974024 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:36.974033 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:36.974039 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:36.974802 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:36.975030 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:36.975900 (+ 870us) server_negotiation.cc:299] Negotiation successful
0620 23:34:36.976523 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":296,"server-negotiator.queue_time_us":2667,"thread_start_us":2366,"threads_started":1}
I20260620 23:34:36.978041 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:36.950865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42019 (local address 127.0.0.1:35714)
0620 23:34:36.952694 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:36.952768 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:36.955186 (+ 2418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:36.956553 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:36.956560 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:36.957351 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:36.957752 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:36.957773 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:36.960401 (+ 2628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:36.960407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:36.961736 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:36.961752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:36.963432 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:36.963438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:36.963668 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:36.964444 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:36.964469 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:36.967272 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:36.972440 (+ 5168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:36.972456 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:36.972473 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:36.972963 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:36.973589 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:36.973597 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:36.973604 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:36.973797 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:36.974987 (+ 1190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:36.974999 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:36.975304 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 23:34:36.975672 (+ 368us) client_negotiation.cc:240] Negotiation successful
0620 23:34:36.976368 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"mutex_wait_us":51,"thread_start_us":1052,"threads_started":1}
I20260620 23:34:36.981866 22908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:34:36.983284 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:36.982691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42252)
0620 23:34:36.982949 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:36.983109 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260620 23:34:36.983659 22858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
I20260620 23:34:36.985780 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:36.985288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42268)
0620 23:34:36.985509 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:36.985653 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:34:36.985993 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:37983: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab.
Entry for principal kudu/127.0.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab.
I20260620 23:34:37.012893 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:37983
--webserver_interface=127.0.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab
--principal=kudu/127.0.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42019,127.0.96.253:37983,127.0.96.252:36353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:34:37.035418 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } attempt: 1
I20260620 23:34:37.037038 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.036445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42284)
0620 23:34:37.036738 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.036858 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 23:34:37.038233 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.037826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42298)
0620 23:34:37.038000 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.038078 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 23:34:37.038561 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:37983: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
I20260620 23:34:37.106868 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } attempt: 2
I20260620 23:34:37.108101 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.107613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42314)
0620 23:34:37.107886 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.107968 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 23:34:37.109675 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.109259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42330)
0620 23:34:37.109443 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.109537 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 23:34:37.110038 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:37983: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
I20260620 23:34:37.201406 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } attempt: 3
I20260620 23:34:37.202996 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.202393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42344)
0620 23:34:37.202689 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.202816 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 23:34:37.204396 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.203948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42356)
0620 23:34:37.204140 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.204225 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 23:34:37.204685 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:37983: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
I20260620 23:34:37.348024 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } attempt: 4
I20260620 23:34:37.349610 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.348902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42372)
0620 23:34:37.349338 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.349449 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250}
I20260620 23:34:37.351023 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.350613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42382)
0620 23:34:37.350778 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.350873 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 23:34:37.351346 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:37983: Network error: Client connection negotiation failed: client connection to 127.0.96.253:37983: connect: Connection refused (error 111)
W20260620 23:34:37.394482 22915 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:34:37.394809 22915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:37.394987 22915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:37.395118 22915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:37.402515 22915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:37.402619 22915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:37.402678 22915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:37.402797 22915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:37.402892 22915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:37.412922 22915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42019,127.0.96.253:37983,127.0.96.252:36353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab
--principal=kudu/127.0.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:37983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:37.414309 22915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:37.415648 22915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:37.429378 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:37.429438 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:37.429435 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:37.434968 22915 server_base.cc:1034] running on GCE node
I20260620 23:34:37.435899 22915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:37.437639 22915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:37.438930 22915 hybrid_clock.cc:648] HybridClock initialized: now 1781998477438853 us; error 63 us; skew 500 ppm
I20260620 23:34:37.439313 22915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:37 dist-test-slave-v34r krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:37.442999 22915 init.cc:377] Logged in from keytab as kudu/127.0.96.253@KRBTEST.COM (short username kudu)
I20260620 23:34:37.444648 22915 webserver.cc:460] Webserver started at http://127.0.96.253:32875/ using document root <none> and password file <none>
I20260620 23:34:37.445487 22915 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:37.445586 22915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:37.445875 22915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:37.449301 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data/instance:
uuid: "94adfd301afb425b8e575364d10b616e"
format_stamp: "Formatted at 2026-06-20 23:34:37 on dist-test-slave-v34r"
server_key: "292a65ca3660feb155c8dfe55bbec42f"
server_key_iv: "960d9913df0a31dd5b63df92ee039929"
server_key_version: "encryptionkey@0"
I20260620 23:34:37.450124 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal/instance:
uuid: "94adfd301afb425b8e575364d10b616e"
format_stamp: "Formatted at 2026-06-20 23:34:37 on dist-test-slave-v34r"
server_key: "292a65ca3660feb155c8dfe55bbec42f"
server_key_iv: "960d9913df0a31dd5b63df92ee039929"
server_key_version: "encryptionkey@0"
I20260620 23:34:37.456225 22915 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260620 23:34:37.460628 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:37.462719 22915 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:37.462993 22915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal
uuid: "94adfd301afb425b8e575364d10b616e"
format_stamp: "Formatted at 2026-06-20 23:34:37 on dist-test-slave-v34r"
server_key: "292a65ca3660feb155c8dfe55bbec42f"
server_key_iv: "960d9913df0a31dd5b63df92ee039929"
server_key_version: "encryptionkey@0"
I20260620 23:34:37.463274 22915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:37.516273 22915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:37.520876 22915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:37.521204 22915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:37.542299 22915 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.253:37983
I20260620 23:34:37.542356 22981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.253:37983 every 8 connection(s)
I20260620 23:34:37.543712 22915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
I20260620 23:34:37.546722 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22915
I20260620 23:34:37.546917 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-1/wal/instance
I20260620 23:34:37.547354 387 external_mini_cluster.cc:1428] Setting key 03004fe01c4ad49b7fe2f5cf7194ee05
I20260620 23:34:37.548774 22982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:37.558630 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:37 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.252@KRBTEST.COM" created.
I20260620 23:34:37.585958 22985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.563138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.0.96.254:42019)
0620 23:34:37.563837 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:37.563848 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:37.564978 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 23:34:37.567449 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:37.567459 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:37.567586 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:37.567768 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:37.569803 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.571140 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.572909 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.573712 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.573902 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:37.577804 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:37.577810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:37.577814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:37.577868 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:37.581487 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:37.582657 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:37.582662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:37.582665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:37.582812 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:37.583427 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:37.583432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:37.583435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:37.583804 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:37.583960 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:37.585220 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0620 23:34:37.585559 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":326,"threads_started":1}
I20260620 23:34:37.587324 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.563807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42019 (local address 127.0.0.1:35718)
0620 23:34:37.564759 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.564821 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:37.566981 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:37.568210 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:37.568224 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:37.568857 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:37.569523 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:37.569540 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.571379 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.571383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:37.572715 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:37.572728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.573990 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.573996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:37.574225 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:37.575004 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:37.575028 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:37.577486 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:37.581700 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:37.581714 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:37.581746 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:37.582482 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:37.582939 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:37.582946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:37.582953 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:37.583219 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:37.583999 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:37.584010 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:37.584582 (+ 572us) client_negotiation.cc:769] Sending connection context
0620 23:34:37.584971 (+ 389us) client_negotiation.cc:240] Negotiation successful
0620 23:34:37.585622 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":439,"threads_started":1}
I20260620 23:34:37.589884 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:37 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:34:37.608582 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.590976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42398)
0620 23:34:37.591371 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.591423 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:37.591647 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:37.592865 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:37.592874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:37.593560 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:37.593836 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:37.593853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.595495 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.595504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:37.597133 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:37.597155 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.598122 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.598131 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:37.598367 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:37.599215 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:37.599246 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:37.602299 (+ 3053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:37.605996 (+ 3697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:37.606003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:37.606007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:37.606375 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:37.606769 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:37.606776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:37.606779 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:37.606918 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:37.607534 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:37.607540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:37.607709 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 23:34:37.607858 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 23:34:37.608185 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260620 23:34:37.608682 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.591366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.0.96.253:37983)
0620 23:34:37.592353 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:37.592360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:37.592387 (+ 27us) server_negotiation.cc:407] Connection header received
0620 23:34:37.592434 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:37.592440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:37.592638 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:37.592768 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:37.594193 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.595298 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.597334 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.597915 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.598024 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:37.602522 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:37.602534 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:37.602550 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:37.602585 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:37.605839 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:37.606520 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:37.606527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:37.606529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:37.606657 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:37.607032 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:37.607038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:37.607046 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:37.607408 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:37.607629 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:37.608020 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 23:34:37.608305 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":826,"thread_start_us":677,"threads_started":1}
I20260620 23:34:37.611238 22982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:34:37.612771 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.612242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40938)
0620 23:34:37.612480 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.612601 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 23:34:37.613253 22931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.615088 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.614659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40944)
0620 23:34:37.614888 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.614978 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:34:37.615407 22982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab.
Entry for principal kudu/127.0.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab.
I20260620 23:34:37.622107 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:36353
--webserver_interface=127.0.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab
--principal=kudu/127.0.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:37565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:42019,127.0.96.253:37983,127.0.96.252:36353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:34:37.650635 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } attempt: 5
Jun 20 23:34:37 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:34:37.664857 22982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 1
I20260620 23:34:37.665202 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.651614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42402)
0620 23:34:37.651907 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.651938 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:37.652125 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:37.652498 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:37.652504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:37.652930 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:37.653178 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:37.653186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.654258 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.654264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:37.655129 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:37.655140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.655895 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.655901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:37.656113 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:37.656898 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:37.656916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:37.659134 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:37.662496 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:37.662502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:37.662506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:37.662896 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:37.663286 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:37.663292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:37.663295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:37.663438 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:37.664066 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:37.664072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:37.664242 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 23:34:37.664399 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 23:34:37.664771 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260620 23:34:37.665259 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.651665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.0.96.253:37983)
0620 23:34:37.651877 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:37.651883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:37.652033 (+ 150us) server_negotiation.cc:407] Connection header received
0620 23:34:37.652269 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:37.652281 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:37.652355 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:37.652466 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:37.653318 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.654127 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.655284 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:37.655731 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:37.655871 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:37.659363 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:37.659368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:37.659371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:37.659405 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:37.662337 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:37.663032 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:37.663039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:37.663047 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:37.663163 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:37.663550 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:37.663557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:37.663560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:37.663938 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:37.664097 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:37.664508 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 23:34:37.664747 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 23:34:37.666314 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.665835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40960)
0620 23:34:37.666046 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.666168 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 23:34:37.667124 22908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:34:37.667861 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.667301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40966)
0620 23:34:37.667554 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.667677 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 23:34:37.668365 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.667910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40982)
0620 23:34:37.668064 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.668161 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 23:34:37.668332 22982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.669482 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.669154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40984)
0620 23:34:37.669308 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.669395 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260620 23:34:37.669740 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.721086 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 1
I20260620 23:34:37.722285 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.721820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40994)
0620 23:34:37.722025 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.722115 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 23:34:37.723232 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.722921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:40996)
0620 23:34:37.723069 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.723133 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:34:37.723448 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.736596 22982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 2
I20260620 23:34:37.738078 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.737511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41006)
0620 23:34:37.737794 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.737906 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 23:34:37.739398 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.738999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41014)
0620 23:34:37.739164 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.739246 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 23:34:37.739956 22982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.791689 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 2
I20260620 23:34:37.792892 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.792411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41026)
0620 23:34:37.792647 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.792735 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:34:37.793913 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.793552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41038)
0620 23:34:37.793748 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.793810 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:34:37.794271 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.831338 22982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 3
I20260620 23:34:37.832902 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.832382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41042)
0620 23:34:37.832640 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.832729 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 23:34:37.834414 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.833914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41054)
0620 23:34:37.834174 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.834277 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260620 23:34:37.834633 22982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.900550 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 3
I20260620 23:34:37.901770 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.901300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41066)
0620 23:34:37.901509 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.901599 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 23:34:37.903276 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.902787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41076)
0620 23:34:37.903074 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.903159 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260620 23:34:37.903818 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:37.980283 22982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 4
I20260620 23:34:37.982123 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.981324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41082)
0620 23:34:37.981677 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.981855 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 23:34:37.983747 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:37.983162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41090)
0620 23:34:37.983408 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:37.983527 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 23:34:37.984115 22982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
W20260620 23:34:38.058055 22990 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:34:38.058573 22990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:38.058712 22990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:38.058859 22990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:38.067845 22990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:38.068029 22990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:38.068215 22990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:38.068325 22990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:38.068447 22990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:38.081094 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 4
I20260620 23:34:38.082741 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.082055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41094)
0620 23:34:38.082375 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.082520 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 23:34:38.084263 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.083740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41098)
0620 23:34:38.083951 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.084064 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260620 23:34:38.084735 22908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:36353: Network error: Client connection negotiation failed: client connection to 127.0.96.252:36353: connect: Connection refused (error 111)
I20260620 23:34:38.086388 22990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:37565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:42019,127.0.96.253:37983,127.0.96.252:36353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab
--principal=kudu/127.0.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:36353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:38.088158 22990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:38.089938 22990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:38.107502 22995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:38.108037 22998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:38.108086 22996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:38.109159 22990 server_base.cc:1034] running on GCE node
I20260620 23:34:38.110263 22990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:38.112576 22990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:38.114104 22990 hybrid_clock.cc:648] HybridClock initialized: now 1781998478113998 us; error 77 us; skew 500 ppm
I20260620 23:34:38.114611 22990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:38.119059 22990 init.cc:377] Logged in from keytab as kudu/127.0.96.252@KRBTEST.COM (short username kudu)
I20260620 23:34:38.121309 22990 webserver.cc:460] Webserver started at http://127.0.96.252:46417/ using document root <none> and password file <none>
I20260620 23:34:38.122262 22990 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:38.122432 22990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:38.122838 22990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:38.126699 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data/instance:
uuid: "d97bf81eefc04e1788028bb03ddace43"
format_stamp: "Formatted at 2026-06-20 23:34:38 on dist-test-slave-v34r"
server_key: "2fc453a07cbe795f4730eb56a5b1c648"
server_key_iv: "0666e56ec3ac4eeaa1fdf3e2cd4d788b"
server_key_version: "encryptionkey@0"
I20260620 23:34:38.127713 22990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal/instance:
uuid: "d97bf81eefc04e1788028bb03ddace43"
format_stamp: "Formatted at 2026-06-20 23:34:38 on dist-test-slave-v34r"
server_key: "2fc453a07cbe795f4730eb56a5b1c648"
server_key_iv: "0666e56ec3ac4eeaa1fdf3e2cd4d788b"
server_key_version: "encryptionkey@0"
I20260620 23:34:38.135031 22990 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260620 23:34:38.140020 23005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:38.142036 22990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 23:34:38.142282 22990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal
uuid: "d97bf81eefc04e1788028bb03ddace43"
format_stamp: "Formatted at 2026-06-20 23:34:38 on dist-test-slave-v34r"
server_key: "2fc453a07cbe795f4730eb56a5b1c648"
server_key_iv: "0666e56ec3ac4eeaa1fdf3e2cd4d788b"
server_key_version: "encryptionkey@0"
I20260620 23:34:38.142586 22990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:38.170889 22990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:38.177420 22990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:38.177793 22990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:38.200873 22990 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.252:36353
I20260620 23:34:38.201053 23056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.252:36353 every 8 connection(s)
I20260620 23:34:38.202409 22990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
I20260620 23:34:38.206475 23057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:38.208323 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 22990
I20260620 23:34:38.208618 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781998087756853-387-0/minicluster-data/master-2/wal/instance
I20260620 23:34:38.209460 387 external_mini_cluster.cc:1428] Setting key 05ee798a569453756d1ac17c8f9bec62
I20260620 23:34:38.216671 23057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
I20260620 23:34:38.247359 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.221880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.0.96.254:42019)
0620 23:34:38.228521 (+ 6641us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.228530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.228564 (+ 34us) server_negotiation.cc:407] Connection header received
0620 23:34:38.229492 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.229499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.229605 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.229736 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.231588 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.232499 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.234406 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.234895 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.235069 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.239812 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.239819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.239823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.239880 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.243544 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.244580 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.244587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.244589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.244759 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.245334 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.245343 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.245346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.245703 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.245910 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.246520 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.246820 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6466,"thread_start_us":6274,"threads_started":1}
I20260620 23:34:38.248634 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.212432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.0.96.252:36353)
0620 23:34:38.227629 (+ 15197us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.227644 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.229237 (+ 1593us) server_negotiation.cc:407] Connection header received
0620 23:34:38.229579 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.229588 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.229942 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.230151 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.231736 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.233123 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.234471 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.235079 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.235304 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.240219 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.240228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.240243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.240285 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.243551 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.244313 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.244320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.244322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.244536 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.244913 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.244919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.244923 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.245582 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.245936 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.246316 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.246895 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":14823,"spinlock_wait_cycles":21376,"thread_start_us":15074,"threads_started":1}
I20260620 23:34:38.248689 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.221205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:42019 (local address 127.0.0.1:35726)
0620 23:34:38.223611 (+ 2406us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.223662 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:38.228977 (+ 5315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:38.229973 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:38.229987 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:38.230811 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:38.231283 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.231306 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.232715 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.232722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.234184 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.234201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.235238 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.235248 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.235439 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.236379 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:38.236407 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:38.239463 (+ 3056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:38.243779 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.243804 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.243818 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.244388 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.244899 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.244907 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.244913 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.245192 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.245884 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:38.245894 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:38.246160 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 23:34:38.246394 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:34:38.247159 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2123,"mutex_wait_us":184,"thread_start_us":1953,"threads_started":1}
I20260620 23:34:38.250968 23057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:34:38.268321 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.251911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42404 (local address 127.0.96.253:37983)
0620 23:34:38.252624 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.252630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.252648 (+ 18us) server_negotiation.cc:407] Connection header received
0620 23:34:38.252711 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.252716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.252796 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.252925 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.254098 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.255112 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.256570 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.257262 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.257374 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.261469 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.261475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.261479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.261515 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.265192 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.266021 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.266026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.266030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.266187 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.266736 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.266762 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.266766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.267071 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.267251 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.267710 (+ 459us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.267975 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":701,"threads_started":1}
I20260620 23:34:38.268388 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.251820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:37983 (local address 127.0.0.1:42404)
0620 23:34:38.252004 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.252025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:38.252118 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:38.253047 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:38.253053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:38.253624 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:38.253879 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.253889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.255257 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.255265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.256416 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.256429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.257427 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.257433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.257640 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.258561 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:38.258589 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:38.261218 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:38.265381 (+ 4163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.265387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.265390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.265873 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.266377 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.266382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.266385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.266604 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.267211 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:38.267219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:38.267408 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 23:34:38.267609 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 23:34:38.268014 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 23:34:38.269979 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.252634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.0.96.254:42019)
0620 23:34:38.252848 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.252852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.252873 (+ 21us) server_negotiation.cc:407] Connection header received
0620 23:34:38.252983 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.252987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.253135 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.253300 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.254661 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.255733 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.257223 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.257841 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.258001 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.262465 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.262474 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.262480 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.262528 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.266211 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.267208 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.267214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.267219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.267369 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.267905 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.267913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.267916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.268355 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.268702 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.269333 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.269622 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 23:34:38.270006 23057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998478, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:34:38.283442 22982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 5
I20260620 23:34:38.285384 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.270831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41108 (local address 127.0.96.252:36353)
0620 23:34:38.271074 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.271079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.271227 (+ 148us) server_negotiation.cc:407] Connection header received
0620 23:34:38.271442 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.271449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.271536 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.271636 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.272653 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.273469 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.274644 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.275055 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.275192 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.278872 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.278877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.278880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.278908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.282046 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.282703 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.282711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.282715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.282831 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.283201 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.283208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.283211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.283597 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.283766 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.284409 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.284886 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":76}
I20260620 23:34:38.285485 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.270816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41108)
0620 23:34:38.271119 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.271143 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:38.271296 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:38.271738 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:38.271741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:38.272287 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:38.272485 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.272509 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.273597 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.273605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.274501 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.274511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.275200 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.275205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.275417 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.276092 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:38.276110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:38.278660 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:38.282186 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.282194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.282198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.282585 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.282947 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.282952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.282955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.283097 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.283852 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:38.283858 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:38.284085 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 23:34:38.284290 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 23:34:38.284683 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260620 23:34:38.289227 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.272425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.0.96.253:37983)
0620 23:34:38.272577 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.272582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.272675 (+ 93us) server_negotiation.cc:407] Connection header received
0620 23:34:38.272898 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.272902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.272968 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.273100 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.274378 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.275079 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.276337 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.276711 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.276851 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.281577 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.281582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.281585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.281615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.285718 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.286766 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.286774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.286781 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.286977 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.287574 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.287578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.287580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.287854 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.288127 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.288476 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.288768 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 23:34:38.293053 23057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: Bootstrap starting.
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
I20260620 23:34:38.296277 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:38.297049 23057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:38.298553 23057 log.cc:824] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:38.302008 23057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: No bootstrap required, opened a new log
I20260620 23:34:38.303623 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.284248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41120)
0620 23:34:38.284565 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.284586 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:38.284714 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:38.286297 (+ 1583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:38.286302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:38.286908 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:38.287180 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.287194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.288849 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.288854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.290302 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.290319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.291315 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.291322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.291591 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.292299 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:38.292323 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:38.295124 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:38.300310 (+ 5186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.300316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.300320 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.300860 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.301450 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.301457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.301463 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.301638 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.302539 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:38.302544 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:38.302729 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 23:34:38.302918 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 23:34:38.303236 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178}
I20260620 23:34:38.303800 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.284326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.96.252:36353)
0620 23:34:38.285905 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.285910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.285927 (+ 17us) server_negotiation.cc:407] Connection header received
0620 23:34:38.286007 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.286013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.286116 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.287389 (+ 1273us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.287586 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.288669 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.290536 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.291100 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.291262 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.295389 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.295398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.295403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.295459 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.300104 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.301063 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.301071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.301077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.301301 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.301768 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.301774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.301778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.302369 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.302575 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.303041 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.303336 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":2545,"threads_started":1}
I20260620 23:34:38.300607 22883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "519d97225ca04828af80fe98cfc325af"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35728
I20260620 23:34:38.310118 23057 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.310488 23057 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97bf81eefc04e1788028bb03ddace43, State: Initialized, Role: FOLLOWER
I20260620 23:34:38.310962 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: Bootstrap starting.
I20260620 23:34:38.310921 23057 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.311827 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.312269 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.312958 23057 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:38.316020 22982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:38.317683 22982 log.cc:824] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:38.321389 22982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: No bootstrap required, opened a new log
W20260620 23:34:38.325137 23078 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:34:38.325428 23078 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:34:38.332655 22982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.333137 22982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94adfd301afb425b8e575364d10b616e, State: Initialized, Role: FOLLOWER
I20260620 23:34:38.333706 22982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.322400 22990 master_runner.cc:186] Error getting master registration for 127.0.96.254:42019: OK, instance_id { permanent_uuid: "519d97225ca04828af80fe98cfc325af" instance_seqno: 1781998476926127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:34:38.335181 22990 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:34:38.335518 23080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.336009 23080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.336262 22982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:34:38.349201 23091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:34:38.349423 23091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:34:38.344601 22915 master_runner.cc:186] Error getting master registration for 127.0.96.254:42019: OK, instance_id { permanent_uuid: "519d97225ca04828af80fe98cfc325af" instance_seqno: 1781998476926127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:34:38.357398 22915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:34:38.362004 22908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } attempt: 5
Jun 20 23:34:38 dist-test-slave-v34r krb5kdc[22828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
I20260620 23:34:38.383657 22911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.362912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:36353 (local address 127.0.0.1:41126)
0620 23:34:38.363099 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:38.363122 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:38.363259 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:38.363828 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:38.363834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:38.364377 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:38.364606 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.364617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.366454 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.366461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.368213 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:38.368243 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.369615 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.369624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:38.369933 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.370844 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:38.370876 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:38.374625 (+ 3749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:38.379400 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.379409 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.379415 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.380148 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.380936 (+ 788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:38.380942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:38.380946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:38.381255 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:38.382267 (+ 1012us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:38.382273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:38.382514 (+ 241us) client_negotiation.cc:769] Sending connection context
0620 23:34:38.382707 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 23:34:38.383105 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 23:34:38.383934 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.363126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41126 (local address 127.0.96.252:36353)
0620 23:34:38.363378 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.363385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.363417 (+ 32us) server_negotiation.cc:407] Connection header received
0620 23:34:38.363506 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.363512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.363614 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.363787 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.364826 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.366184 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.368472 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.369341 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.369505 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.374963 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.374971 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.374976 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.375031 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.379115 (+ 4084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.380384 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.380390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.380394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.380714 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.381435 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.381442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.381445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.382060 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.382317 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.382873 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.383273 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260620 23:34:38.390683 22908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af: Bootstrap starting.
I20260620 23:34:38.394827 22908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:38.396194 22908 log.cc:824] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:38.400919 22908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af: No bootstrap required, opened a new log
I20260620 23:34:38.410231 22908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.410827 22908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519d97225ca04828af80fe98cfc325af, State: Initialized, Role: FOLLOWER
I20260620 23:34:38.411505 22908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.418079 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.418985 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.419768 22908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:38.419766 22883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:34:38.420231 22883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:38.423813 22883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.426323 22957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "519d97225ca04828af80fe98cfc325af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "94adfd301afb425b8e575364d10b616e"
I20260620 23:34:38.426841 22957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:38.430702 22957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519d97225ca04828af80fe98cfc325af in term 1.
W20260620 23:34:38.430934 23011 tablet.cc:2369] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:34:38.431747 22858 leader_election.cc:302] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519d97225ca04828af80fe98cfc325af, 94adfd301afb425b8e575364d10b616e; no voters:
I20260620 23:34:38.433544 23032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "519d97225ca04828af80fe98cfc325af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d97bf81eefc04e1788028bb03ddace43"
I20260620 23:34:38.434067 23032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:38.437778 23032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 519d97225ca04828af80fe98cfc325af in term 1.
I20260620 23:34:38.438544 22883 leader_election.cc:288] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [CANDIDATE]: Term 1 election: Requested vote from peers 94adfd301afb425b8e575364d10b616e (127.0.96.253:37983), d97bf81eefc04e1788028bb03ddace43 (127.0.96.252:36353)
I20260620 23:34:38.445611 23094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:38.457474 23094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [term 1 LEADER]: Becoming Leader. State: Replica: 519d97225ca04828af80fe98cfc325af, State: Running, Role: LEADER
I20260620 23:34:38.458391 23094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } }
I20260620 23:34:38.473547 23097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519d97225ca04828af80fe98cfc325af. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.474129 23097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:38.490372 23107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:38.508327 23107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:38.511416 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.465926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.96.253:37983)
0620 23:34:38.466212 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.466221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.479168 (+ 12947us) server_negotiation.cc:407] Connection header received
0620 23:34:38.480252 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.480259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.480370 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.480464 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.485993 (+ 5529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.487188 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.492721 (+ 5533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.493434 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.493540 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.500383 (+ 6843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.500393 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.500399 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.500475 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.506625 (+ 6150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.507799 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.507806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.507809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.508040 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.508582 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.508588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.508591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.509053 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.509382 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.510628 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.510916 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 23:34:38.520305 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.466409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41140 (local address 127.0.96.252:36353)
0620 23:34:38.466620 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.466629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.470797 (+ 4168us) server_negotiation.cc:407] Connection header received
0620 23:34:38.472765 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.472775 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.472898 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.473020 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.480234 (+ 7214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.481633 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.486715 (+ 5082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.487830 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.488023 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.497919 (+ 9896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.497929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.497935 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.498009 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.504552 (+ 6543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.517186 (+ 12634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.517195 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.517199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.517459 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.518024 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.518029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.518033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.518497 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.518770 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.519448 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.519832 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 23:34:38.524729 22936 tablet.cc:2369] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:34:38.530817 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:38.467097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.0.96.254:42019)
0620 23:34:38.467408 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.467417 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.479939 (+ 12522us) server_negotiation.cc:407] Connection header received
0620 23:34:38.481455 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.481461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.481548 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.481619 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.488971 (+ 7352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.490280 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.496938 (+ 6658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:38.497815 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:38.497930 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:38.522629 (+ 24699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:38.522637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:38.522642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:38.522698 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:38.526816 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.527949 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.527956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.527960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.528136 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:38.528594 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:38.528599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:38.528603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:38.529113 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:38.529411 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:38.530052 (+ 641us) server_negotiation.cc:299] Negotiation successful
0620 23:34:38.530353 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":116}
I20260620 23:34:38.544873 23032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [term 1 FOLLOWER]: Refusing update from remote peer 519d97225ca04828af80fe98cfc325af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:34:38.547062 23094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:34:38.556406 22957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [term 1 FOLLOWER]: Refusing update from remote peer 519d97225ca04828af80fe98cfc325af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:34:38.559286 23094 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [LEADER]: Connected to new peer: Peer: permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:34:38.606076 23080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519d97225ca04828af80fe98cfc325af. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.606480 23080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.620450 23112 mvcc.cc:205] Tried to move back new op lower bound from 7299065768088862720 to 7299065767782678528. Current Snapshot: MvccSnapshot[applied={T|T < 7299065768088862720}]
I20260620 23:34:38.626861 23115 mvcc.cc:205] Tried to move back new op lower bound from 7299065768088862720 to 7299065767782678528. Current Snapshot: MvccSnapshot[applied={T|T < 7299065768088862720}]
I20260620 23:34:38.628643 23080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.629030 23080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.629371 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519d97225ca04828af80fe98cfc325af. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.629678 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.631603 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.633108 23107 catalog_manager.cc:1347] Generated new cluster ID: f602e5f88cde448c8470a9207f970f32
I20260620 23:34:38.649871 23107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:38.661145 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:38.661738 23094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.662590 23094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:38.687722 23114 mvcc.cc:205] Tried to move back new op lower bound from 7299065768088862720 to 7299065767782678528. Current Snapshot: MvccSnapshot[applied={T|T < 7299065768088862720}]
I20260620 23:34:38.694926 23066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "519d97225ca04828af80fe98cfc325af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519d97225ca04828af80fe98cfc325af" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 42019 } } peers { permanent_uuid: "94adfd301afb425b8e575364d10b616e" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 37983 } } peers { permanent_uuid: "d97bf81eefc04e1788028bb03ddace43" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 36353 } } }
I20260620 23:34:38.695319 23066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:38.714792 23107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:38.719193 23107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:38.775483 23107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519d97225ca04828af80fe98cfc325af: Generated new TSK 0
I20260620 23:34:38.777113 23107 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:34:38.965929 23062 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:38.929374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.0.96.254:42019)
0620 23:34:38.929654 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.929662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.960846 (+ 31184us) server_negotiation.cc:407] Connection header received
0620 23:34:38.964154 (+ 3308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.964161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.964252 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.964330 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.965638 (+ 1308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35746: BlockingRecv error: recv got EOF from 127.0.0.1:35746 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 23:34:38.970162 23060 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:38.942874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.0.96.252:36353)
0620 23:34:38.943152 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.943162 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.967492 (+ 24330us) server_negotiation.cc:407] Connection header received
0620 23:34:38.968436 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.968444 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.968557 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.968644 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.969839 (+ 1195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41152: BlockingRecv error: recv got EOF from 127.0.0.1:41152 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 23:34:38.971292 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:38.931692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.96.253:37983)
0620 23:34:38.931989 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:38.931999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:38.963435 (+ 31436us) server_negotiation.cc:407] Connection header received
0620 23:34:38.963714 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:38.963721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:38.963833 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:38.963961 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:38.971014 (+ 7053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42440: BlockingRecv error: recv got EOF from 127.0.0.1:42440 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 23:34:39.007812 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:38.999865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.0.96.253:37983)
0620 23:34:39.000180 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.000191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.005908 (+ 5717us) server_negotiation.cc:407] Connection header received
0620 23:34:39.006238 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.006244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.006376 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.006513 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.007541 (+ 1028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42444: BlockingRecv error: recv got EOF from 127.0.0.1:42444 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 23:34:39.027722 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.018801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.0.96.253:37983)
0620 23:34:39.019069 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.019081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.025918 (+ 6837us) server_negotiation.cc:407] Connection header received
0620 23:34:39.026219 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.026224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.026320 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.026451 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.027423 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42450: BlockingRecv error: recv got EOF from 127.0.0.1:42450 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 23:34:39.036160 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.033448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.0.96.253:37983)
0620 23:34:39.033740 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.033750 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.034326 (+ 576us) server_negotiation.cc:407] Connection header received
0620 23:34:39.034588 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.034601 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.034690 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.034812 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.035680 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42458: BlockingRecv error: recv got EOF from 127.0.0.1:42458 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260620 23:34:39.053236 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.050470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.0.96.253:37983)
0620 23:34:39.050833 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.050844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.051469 (+ 625us) server_negotiation.cc:407] Connection header received
0620 23:34:39.051798 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.051809 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.051896 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.052020 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.052900 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42466: BlockingRecv error: recv got EOF from 127.0.0.1:42466 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260620 23:34:39.071980 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.069459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.96.253:37983)
0620 23:34:39.069723 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.069729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.070105 (+ 376us) server_negotiation.cc:407] Connection header received
0620 23:34:39.070320 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.070326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.070447 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.070619 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.071670 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42480: BlockingRecv error: recv got EOF from 127.0.0.1:42480 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 23:34:39.094206 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.091977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.0.96.253:37983)
0620 23:34:39.092223 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.092229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.092596 (+ 367us) server_negotiation.cc:407] Connection header received
0620 23:34:39.092794 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.092800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.092910 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.093111 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.093948 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42488: BlockingRecv error: recv got EOF from 127.0.0.1:42488 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 23:34:39.132690 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.130488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42498 (local address 127.0.96.253:37983)
0620 23:34:39.130710 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.130717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.131176 (+ 459us) server_negotiation.cc:407] Connection header received
0620 23:34:39.131380 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.131386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.131489 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.131639 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.132433 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42498: BlockingRecv error: recv got EOF from 127.0.0.1:42498 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 23:34:39.204020 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.201466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42504 (local address 127.0.96.253:37983)
0620 23:34:39.201785 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.201793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.202272 (+ 479us) server_negotiation.cc:407] Connection header received
0620 23:34:39.202516 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.202521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.202618 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.202738 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.203711 (+ 973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42504: BlockingRecv error: recv got EOF from 127.0.0.1:42504 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 23:34:39.327951 23078 catalog_manager.cc:1259] Loaded cluster ID: f602e5f88cde448c8470a9207f970f32
I20260620 23:34:39.328099 23078 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: loading cluster ID for follower catalog manager: success
I20260620 23:34:39.331168 23078 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: acquiring CA information for follower catalog manager: success
I20260620 23:34:39.333523 23078 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d97bf81eefc04e1788028bb03ddace43: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:34:39.340678 23063 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:39.338161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.0.96.253:37983)
0620 23:34:39.338464 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.338469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.338664 (+ 195us) server_negotiation.cc:407] Connection header received
0620 23:34:39.338996 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.339002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.339110 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.339235 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:39.340351 (+ 1116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42518: BlockingRecv error: recv got EOF from 127.0.0.1:42518 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 23:34:39.352029 23091 catalog_manager.cc:1259] Loaded cluster ID: f602e5f88cde448c8470a9207f970f32
I20260620 23:34:39.352259 23091 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: loading cluster ID for follower catalog manager: success
I20260620 23:34:39.355671 23091 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: acquiring CA information for follower catalog manager: success
I20260620 23:34:39.358405 23091 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94adfd301afb425b8e575364d10b616e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:34:39.608597 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:39.602623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.0.96.253:37983)
0620 23:34:39.602846 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.602852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.603026 (+ 174us) server_negotiation.cc:407] Connection header received
0620 23:34:39.603138 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.603143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.603369 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.603532 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:39.604003 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.604814 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.606211 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.606660 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.606786 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:39.607260 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:39.607946 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:39.608078 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:39.608214 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 23:34:39.608302 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 23:34:39.622623 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:39.615761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.0.96.252:36353)
0620 23:34:39.616370 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.616377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.616403 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:34:39.616766 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.616770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.616963 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.617235 (+ 272us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:39.617682 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.618487 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.619925 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.620357 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.620563 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:39.621164 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:39.621893 (+ 729us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:39.622060 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:39.622127 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 23:34:39.622229 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":268,"threads_started":1}
I20260620 23:34:39.641690 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:39.631616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.0.96.252:36353)
0620 23:34:39.632209 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.632219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.634862 (+ 2643us) server_negotiation.cc:407] Connection header received
0620 23:34:39.635124 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.635131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.635261 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.635353 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:39.636151 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.637359 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.639501 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.640112 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.640276 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:39.640803 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:39.640958 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:39.641066 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:39.641319 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 23:34:39.641413 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260620 23:34:39.643676 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:39.631895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.0.96.253:37983)
0620 23:34:39.632417 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.632426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.636870 (+ 4444us) server_negotiation.cc:407] Connection header received
0620 23:34:39.637124 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.637130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.637252 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.637391 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:39.638121 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.639360 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.641407 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.641964 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.642128 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:39.642698 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:39.642858 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:39.642968 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:39.643178 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 23:34:39.643283 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260620 23:34:39.644843 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:39.631092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.0.96.254:42019)
0620 23:34:39.635491 (+ 4399us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:39.635497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:39.635516 (+ 19us) server_negotiation.cc:407] Connection header received
0620 23:34:39.635641 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:39.635646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:39.635883 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:39.636084 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:39.637167 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.638220 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.640621 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:39.641276 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:39.641407 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:39.643174 (+ 1767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:39.643700 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:39.643811 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:39.644430 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 23:34:39.644530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4257,"thread_start_us":4133,"threads_started":1}
I20260620 23:34:39.650264 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22841
I20260620 23:34:39.651760 22903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:39.780238 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22841
W20260620 23:34:39.803017 23007 connection.cc:535] server connection from 127.0.0.1:41126 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:34:39.803620 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22915
I20260620 23:34:39.805160 22977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:39.932060 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22915
I20260620 23:34:39.954281 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 22990
I20260620 23:34:39.955552 23052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:40.086149 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 22990
2026-06-20T23:34:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5177 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 23:34:40 dist-test-slave-v34r krb5kdc[23206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 23:34:40 dist-test-slave-v34r krb5kdc[23206](info): set up 2 sockets
Jun 20 23:34:40 dist-test-slave-v34r krb5kdc[23206](info): commencing operation
krb5kdc: starting...
W20260620 23:34:41.237777 387 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/test-user.keytab.
Jun 20 23:34:41 dist-test-slave-v34r krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T23:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T23:34:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.96.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
Entry for principal kudu/127.0.96.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab.
I20260620 23:34:41.548126 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:37163
--webserver_interface=127.0.96.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:37163,127.0.96.253:41349,127.0.96.252:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 23:34:41.913380 23219 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:34:41.913726 23219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:41.913794 23219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:41.913873 23219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:41.921437 23219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:41.921545 23219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:41.921604 23219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:41.921649 23219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:41.921715 23219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:41.932200 23219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:37163,127.0.96.253:41349,127.0.96.252:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.254.keytab
--principal=kudu/127.0.96.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.254:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.96.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:41.933562 23219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:41.934887 23219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:41.948678 23224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:41.948873 23227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:41.949463 23225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:41.950033 23219 server_base.cc:1034] running on GCE node
I20260620 23:34:41.950919 23219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:41.952538 23219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:41.953830 23219 hybrid_clock.cc:648] HybridClock initialized: now 1781998481953711 us; error 97 us; skew 500 ppm
I20260620 23:34:41.954248 23219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:41 dist-test-slave-v34r krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:41.961470 23219 init.cc:377] Logged in from keytab as kudu/127.0.96.254@KRBTEST.COM (short username kudu)
I20260620 23:34:41.963352 23219 webserver.cc:460] Webserver started at http://127.0.96.254:41819/ using document root <none> and password file <none>
I20260620 23:34:41.964141 23219 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:41.964239 23219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:41.964610 23219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:41.967850 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data/instance:
uuid: "aca7efb8ec6f45d7bd95492115fe8fc9"
format_stamp: "Formatted at 2026-06-20 23:34:41 on dist-test-slave-v34r"
server_key: "14fd796d27a37244099d214adfaacd1c"
server_key_iv: "763357bfc4b43b763f3c55b88a32604a"
server_key_version: "encryptionkey@0"
I20260620 23:34:41.968705 23219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal/instance:
uuid: "aca7efb8ec6f45d7bd95492115fe8fc9"
format_stamp: "Formatted at 2026-06-20 23:34:41 on dist-test-slave-v34r"
server_key: "14fd796d27a37244099d214adfaacd1c"
server_key_iv: "763357bfc4b43b763f3c55b88a32604a"
server_key_version: "encryptionkey@0"
I20260620 23:34:41.974745 23219 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260620 23:34:41.979079 23234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:41.980873 23219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 23:34:41.981155 23219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal
uuid: "aca7efb8ec6f45d7bd95492115fe8fc9"
format_stamp: "Formatted at 2026-06-20 23:34:41 on dist-test-slave-v34r"
server_key: "14fd796d27a37244099d214adfaacd1c"
server_key_iv: "763357bfc4b43b763f3c55b88a32604a"
server_key_version: "encryptionkey@0"
I20260620 23:34:41.981417 23219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:42.025635 23219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:42.030499 23219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:42.030925 23219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:42.052922 23219 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.254:37163
I20260620 23:34:42.053004 23285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.254:37163 every 8 connection(s)
I20260620 23:34:42.054436 23219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/data/info.pb
I20260620 23:34:42.059501 23286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:42.062378 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 23219
I20260620 23:34:42.062541 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-0/wal/instance
I20260620 23:34:42.063090 387 external_mini_cluster.cc:1428] Setting key 3ed753470d89586e23b70b60f580e736
I20260620 23:34:42.068199 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.96.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.253@KRBTEST.COM" created.
Jun 20 23:34:42 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:34:42.099835 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.072488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57288 (local address 127.0.96.254:37163)
0620 23:34:42.073647 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:42.073669 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:42.073915 (+ 246us) server_negotiation.cc:407] Connection header received
0620 23:34:42.077061 (+ 3146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:42.077078 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:42.077410 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:42.077672 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:42.079259 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.080670 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.082312 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.082946 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.083171 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.090093 (+ 6922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:42.090111 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:42.090136 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:42.090212 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:42.094598 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.095666 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.095679 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.095685 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.095941 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.096407 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.096414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.096420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.097278 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:42.097517 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:42.098232 (+ 715us) server_negotiation.cc:299] Negotiation successful
0620 23:34:42.098589 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":867,"thread_start_us":548,"threads_started":1}
I20260620 23:34:42.099895 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.072471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37163 (local address 127.0.0.1:57288)
0620 23:34:42.073700 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.073766 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:42.076436 (+ 2670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:42.077683 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:42.077690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:42.078577 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:42.079007 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.079028 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.080847 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.080853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.082144 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.082156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.083218 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.083224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.083404 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.084206 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:42.084251 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:42.089638 (+ 5387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:42.094852 (+ 5214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.094868 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.094883 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.095439 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.096073 (+ 634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.096076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.096078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.096271 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.097431 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:42.097438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:42.097708 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 23:34:42.098094 (+ 386us) client_negotiation.cc:240] Negotiation successful
0620 23:34:42.098836 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":66,"thread_start_us":794,"threads_started":1}
I20260620 23:34:42.103132 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:34:42.104561 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.103906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41234)
0620 23:34:42.104167 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.104349 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 23:34:42.104945 23237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
I20260620 23:34:42.107013 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.106374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41250)
0620 23:34:42.106622 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.106749 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:34:42.107302 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:41349: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab.
Entry for principal kudu/127.0.96.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab.
I20260620 23:34:42.126543 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:41349
--webserver_interface=127.0.96.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab
--principal=kudu/127.0.96.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:37163,127.0.96.253:41349,127.0.96.252:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:34:42.156805 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } attempt: 1
I20260620 23:34:42.158490 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.157783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41252)
0620 23:34:42.158105 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.158292 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 23:34:42.159780 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.159288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41254)
0620 23:34:42.159451 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.159551 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 23:34:42.160035 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:41349: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
I20260620 23:34:42.228415 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } attempt: 2
I20260620 23:34:42.230005 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.229227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41260)
0620 23:34:42.229579 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.229772 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260620 23:34:42.231549 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.231019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41266)
0620 23:34:42.231211 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.231343 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:34:42.231877 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:41349: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
I20260620 23:34:42.323212 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } attempt: 3
I20260620 23:34:42.324707 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.324014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41268)
0620 23:34:42.324355 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.324492 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260620 23:34:42.326226 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.325762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41272)
0620 23:34:42.325957 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.326045 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260620 23:34:42.326539 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:41349: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
I20260620 23:34:42.469882 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } attempt: 4
I20260620 23:34:42.471477 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.470686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41274)
0620 23:34:42.471161 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.471294 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241}
I20260620 23:34:42.472847 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.472429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41282)
0620 23:34:42.472606 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.472699 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260620 23:34:42.473170 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.253:41349: Network error: Client connection negotiation failed: client connection to 127.0.96.253:41349: connect: Connection refused (error 111)
W20260620 23:34:42.496366 23293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:34:42.496745 23293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:42.496896 23293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:42.497076 23293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:42.504549 23293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:42.504664 23293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:42.504724 23293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:42.504820 23293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:42.504933 23293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:42.515826 23293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:37163,127.0.96.253:41349,127.0.96.252:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.253.keytab
--principal=kudu/127.0.96.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.253:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.96.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:42.517328 23293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:42.518729 23293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:42.531569 23298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:42.531824 23301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:42.532024 23299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:42.534791 23293 server_base.cc:1034] running on GCE node
I20260620 23:34:42.535933 23293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:42.538002 23293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:42.539263 23293 hybrid_clock.cc:648] HybridClock initialized: now 1781998482539178 us; error 45 us; skew 500 ppm
I20260620 23:34:42.539678 23293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:42 dist-test-slave-v34r krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:42.543969 23293 init.cc:377] Logged in from keytab as kudu/127.0.96.253@KRBTEST.COM (short username kudu)
I20260620 23:34:42.546094 23293 webserver.cc:460] Webserver started at http://127.0.96.253:46463/ using document root <none> and password file <none>
I20260620 23:34:42.547106 23293 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:42.547242 23293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:42.547585 23293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:42.551467 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data/instance:
uuid: "95f26e354e43414fa257d247e5f954f9"
format_stamp: "Formatted at 2026-06-20 23:34:42 on dist-test-slave-v34r"
server_key: "d53e747d07b23e87e8b2777b381a3d0a"
server_key_iv: "948ed21725198c1b534969332b613bb4"
server_key_version: "encryptionkey@0"
I20260620 23:34:42.552517 23293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal/instance:
uuid: "95f26e354e43414fa257d247e5f954f9"
format_stamp: "Formatted at 2026-06-20 23:34:42 on dist-test-slave-v34r"
server_key: "d53e747d07b23e87e8b2777b381a3d0a"
server_key_iv: "948ed21725198c1b534969332b613bb4"
server_key_version: "encryptionkey@0"
I20260620 23:34:42.559964 23293 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260620 23:34:42.565690 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:42.567883 23293 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260620 23:34:42.568171 23293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal
uuid: "95f26e354e43414fa257d247e5f954f9"
format_stamp: "Formatted at 2026-06-20 23:34:42 on dist-test-slave-v34r"
server_key: "d53e747d07b23e87e8b2777b381a3d0a"
server_key_iv: "948ed21725198c1b534969332b613bb4"
server_key_version: "encryptionkey@0"
I20260620 23:34:42.568491 23293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:42.586210 23293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:42.592370 23293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:42.592769 23293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:42.616921 23293 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.253:41349
I20260620 23:34:42.616959 23359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.253:41349 every 8 connection(s)
I20260620 23:34:42.618937 23293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/data/info.pb
I20260620 23:34:42.623288 23360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:42.629254 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 23293
I20260620 23:34:42.629468 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-1/wal/instance
I20260620 23:34:42.630033 387 external_mini_cluster.cc:1428] Setting key ff145e572d9814adc2985d5112301720
I20260620 23:34:42.630596 23360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:42 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.96.252@KRBTEST.COM; defaulting to no policy
I20260620 23:34:42.659914 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.634413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57294 (local address 127.0.96.254:37163)
0620 23:34:42.635072 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:42.635082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:42.636428 (+ 1346us) server_negotiation.cc:407] Connection header received
0620 23:34:42.638841 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:42.638851 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:42.638970 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:42.639155 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:42.642204 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.643808 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.645859 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.646718 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.646899 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.651825 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:42.651836 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:42.651845 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:42.651934 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:42.655801 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.656846 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.656851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.656854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.657059 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.657553 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.657559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.657562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.657906 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:42.658111 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:42.659069 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 23:34:42.659353 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":314,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.96.252@KRBTEST.COM" created.
I20260620 23:34:42.661458 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.635151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37163 (local address 127.0.0.1:57294)
0620 23:34:42.636188 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.636243 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:42.638468 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:42.639590 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:42.639608 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:42.640565 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:42.641696 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.641733 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.644067 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.644073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.645664 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.645681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.647016 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.647022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.647288 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.648245 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:42.648284 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:42.651450 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:42.656042 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.656056 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.656077 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.656673 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.657186 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.657198 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.657202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.657415 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.658050 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:42.658063 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:42.658544 (+ 481us) client_negotiation.cc:769] Sending connection context
0620 23:34:42.658948 (+ 404us) client_negotiation.cc:240] Negotiation successful
0620 23:34:42.659814 (+ 866us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":437,"threads_started":1}
I20260620 23:34:42.663810 23360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:42 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:34:42.683002 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.664875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41294)
0620 23:34:42.665155 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.665179 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:42.665327 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:42.666655 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:42.666659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:42.667128 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:42.667312 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.667320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.669543 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.669548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.670504 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.670517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.671928 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.671937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.672197 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.673077 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:42.673108 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:42.676278 (+ 3170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:42.680168 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.680176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.680183 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.680593 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.681078 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.681082 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.681084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.681232 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.681904 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:42.681908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:42.682070 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 23:34:42.682214 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 23:34:42.682563 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 23:34:42.683135 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.665125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.0.96.253:41349)
0620 23:34:42.666027 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:42.666038 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:42.666077 (+ 39us) server_negotiation.cc:407] Connection header received
0620 23:34:42.666156 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:42.666167 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:42.666460 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:42.666645 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:42.667511 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.669354 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.670679 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.671671 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.671817 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.676506 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:42.676517 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:42.676530 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:42.676560 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:42.680002 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.680741 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.680758 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.680761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.680886 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.681359 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.681368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.681371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.681763 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:42.682004 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:42.682404 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 23:34:42.682694 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":670,"thread_start_us":476,"threads_started":1}
I20260620 23:34:42.686414 23360 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:34:42.687911 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.687349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41616)
0620 23:34:42.687565 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.687746 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 23:34:42.688226 23312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:42.690086 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.689717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41632)
0620 23:34:42.689883 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.689974 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260620 23:34:42.690356 23360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.96.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab.
Entry for principal kudu/127.0.96.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab.
I20260620 23:34:42.700455 387 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
/tmp/dist-test-taskWbSr8K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:37881
--webserver_interface=127.0.96.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab
--principal=kudu/127.0.96.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.96.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.96.254:37163,127.0.96.253:41349,127.0.96.252:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 23:34:42.739714 23360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 1
I20260620 23:34:42.741391 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.740732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41642)
0620 23:34:42.741069 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.741204 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 23:34:42.742600 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.742243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41656)
0620 23:34:42.742407 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.742493 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 23:34:42.742830 23360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:42.772529 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } attempt: 5
Jun 20 23:34:42 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:34:42.790086 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.773647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.0.96.253:41349)
0620 23:34:42.773929 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:42.773935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:42.773987 (+ 52us) server_negotiation.cc:407] Connection header received
0620 23:34:42.774241 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:42.774246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:42.774338 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:42.774436 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:42.775647 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.776402 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.778263 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.778722 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.778873 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.783264 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:42.783269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:42.783273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:42.783315 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:42.786857 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.787985 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.787990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.787993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.788129 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:42.788581 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:42.788584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:42.788586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:42.788898 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:42.789092 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:42.789619 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 23:34:42.789853 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 23:34:42.790491 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.773572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41296)
0620 23:34:42.773867 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.773896 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:42.774087 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:42.774519 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:42.774525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:42.775145 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:42.775445 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.775458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.776588 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.776595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.778063 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:42.778081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:42.778913 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:42.778919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:42.779178 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:42.780007 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:42.780033 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:42.782974 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:42.787062 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.787071 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.787078 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.787815 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.788259 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:42.788264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:42.788268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:42.788456 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:42.789051 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:42.789056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:42.789254 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 23:34:42.789488 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 23:34:42.789966 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 23:34:42.792022 23286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } has no permanent_uuid. Determining permanent_uuid...
I20260620 23:34:42.793565 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.792953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41658)
0620 23:34:42.793229 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.793352 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 23:34:42.794752 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.794342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41662)
0620 23:34:42.794502 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.794622 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 23:34:42.795035 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:42.811120 23360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 2
I20260620 23:34:42.812379 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.811903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41666)
0620 23:34:42.812108 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.812207 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 23:34:42.813916 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.813491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41680)
0620 23:34:42.813677 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.813797 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260620 23:34:42.814311 23360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:42.846367 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 1
I20260620 23:34:42.847983 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.847322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41688)
0620 23:34:42.847605 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.847781 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 23:34:42.849275 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.848865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41694)
0620 23:34:42.849089 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.849164 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 23:34:42.849551 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:42.905575 23360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 3
I20260620 23:34:42.907101 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.906476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41704)
0620 23:34:42.906815 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.906940 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 23:34:42.908562 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.908091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41712)
0620 23:34:42.908356 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.908445 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
W20260620 23:34:42.908771 23360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:42.917850 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 2
I20260620 23:34:42.919368 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.918780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41716)
0620 23:34:42.919082 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.919192 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260620 23:34:42.920447 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:42.920128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41724)
0620 23:34:42.920274 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:42.920350 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 23:34:42.920662 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:43.026934 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 3
I20260620 23:34:43.028481 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.027888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41738)
0620 23:34:43.028181 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.028303 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260620 23:34:43.029644 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.029250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41744)
0620 23:34:43.029483 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.029549 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260620 23:34:43.029932 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:43.052038 23360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 4
I20260620 23:34:43.053359 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.052822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41750)
0620 23:34:43.053089 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.053200 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 23:34:43.054351 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.054038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41760)
0620 23:34:43.054186 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.054251 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:34:43.054589 23360 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
W20260620 23:34:43.129566 23368 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 23:34:43.130064 23368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 23:34:43.130198 23368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 23:34:43.130352 23368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 23:34:43.138971 23368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 23:34:43.139114 23368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 23:34:43.139218 23368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 23:34:43.139328 23368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 23:34:43.139454 23368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 23:34:43.151603 23368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.96.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.96.254:37163,127.0.96.253:41349,127.0.96.252:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/krb5kdc/kudu_127.0.96.252.keytab
--principal=kudu/127.0.96.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.96.252:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.96.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 22:07:19 UTC on ccycloud.quasar-uhqtot.root.comops.site
build id 4457790
ASAN enabled
I20260620 23:34:43.153503 23368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 23:34:43.155273 23368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 23:34:43.170663 23373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:43.170823 23374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 23:34:43.171512 23376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 23:34:43.174834 23368 server_base.cc:1034] running on GCE node
I20260620 23:34:43.175977 23368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 23:34:43.177990 23368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 23:34:43.179288 23368 hybrid_clock.cc:648] HybridClock initialized: now 1781998483179213 us; error 56 us; skew 500 ppm
I20260620 23:34:43.179759 23368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 23:34:43.184070 23368 init.cc:377] Logged in from keytab as kudu/127.0.96.252@KRBTEST.COM (short username kudu)
I20260620 23:34:43.186478 23368 webserver.cc:460] Webserver started at http://127.0.96.252:44839/ using document root <none> and password file <none>
I20260620 23:34:43.187407 23368 fs_manager.cc:359] Metadata directory not provided
I20260620 23:34:43.187520 23368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 23:34:43.187844 23368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 23:34:43.191675 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data/instance:
uuid: "554489a6b6964aaa8869a29ad8e78c0a"
format_stamp: "Formatted at 2026-06-20 23:34:43 on dist-test-slave-v34r"
server_key: "937b1cf915acb4eec6277ef5c08593a6"
server_key_iv: "c2504f27e111113979d70e0646f1cdf1"
server_key_version: "encryptionkey@0"
I20260620 23:34:43.192667 23368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal/instance:
uuid: "554489a6b6964aaa8869a29ad8e78c0a"
format_stamp: "Formatted at 2026-06-20 23:34:43 on dist-test-slave-v34r"
server_key: "937b1cf915acb4eec6277ef5c08593a6"
server_key_iv: "c2504f27e111113979d70e0646f1cdf1"
server_key_version: "encryptionkey@0"
I20260620 23:34:43.199738 23368 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 23:34:43.204448 23383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 23:34:43.206475 23368 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 23:34:43.206703 23368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data,/tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal
uuid: "554489a6b6964aaa8869a29ad8e78c0a"
format_stamp: "Formatted at 2026-06-20 23:34:43 on dist-test-slave-v34r"
server_key: "937b1cf915acb4eec6277ef5c08593a6"
server_key_iv: "c2504f27e111113979d70e0646f1cdf1"
server_key_version: "encryptionkey@0"
I20260620 23:34:43.206950 23368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 23:34:43.207134 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 4
I20260620 23:34:43.208209 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.207786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41768)
0620 23:34:43.207977 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.208069 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260620 23:34:43.209219 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.208890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41778)
0620 23:34:43.209057 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.209119 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 23:34:43.209468 23286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.96.252:37881: Network error: Client connection negotiation failed: client connection to 127.0.96.252:37881: connect: Connection refused (error 111)
I20260620 23:34:43.236063 23368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 23:34:43.241737 23368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 23:34:43.242100 23368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 23:34:43.265239 23368 rpc_server.cc:307] RPC server started. Bound to: 127.0.96.252:37881
I20260620 23:34:43.265298 23434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.96.252:37881 every 8 connection(s)
I20260620 23:34:43.266968 23368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/data/info.pb
I20260620 23:34:43.271101 23435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 23:34:43.272856 387 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu as pid 23368
I20260620 23:34:43.273099 387 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWbSr8K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781998087756853-387-0/minicluster-data/master-2/wal/instance
I20260620 23:34:43.273669 387 external_mini_cluster.cc:1428] Setting key b95136d33f869ec4ec0d54dfeaafb98c
I20260620 23:34:43.279763 23435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:34:43.301457 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.283240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.0.96.254:37163)
0620 23:34:43.283776 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.283783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.284428 (+ 645us) server_negotiation.cc:407] Connection header received
0620 23:34:43.284884 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.284889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.285039 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.285158 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.286715 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.287807 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.289996 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.290475 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.290644 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.294643 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.294649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.294653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.294690 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.298147 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.299109 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.299114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.299116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.299282 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.299752 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.299760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.299766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.300144 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.300323 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.300812 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.301103 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":273,"threads_started":1}
I20260620 23:34:43.302102 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.275983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.0.96.252:37881)
0620 23:34:43.277107 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.277121 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.277187 (+ 66us) server_negotiation.cc:407] Connection header received
0620 23:34:43.279212 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.279242 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.279809 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.280259 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.282172 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.284384 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.286294 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.287343 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.287549 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.292558 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.292569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.292585 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.292626 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.296350 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.297655 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.297663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.297671 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.297947 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.298554 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.298558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.298562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.299292 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.299642 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.300121 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.300588 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":486,"threads_started":1}
I20260620 23:34:43.302134 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.283221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.254:37163 (local address 127.0.0.1:57308)
0620 23:34:43.284252 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.284348 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:43.284574 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:43.285271 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:43.285278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:43.286194 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:43.286550 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.286561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.287992 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.287996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.289789 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.289808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.290766 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.290776 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.290927 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.291631 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:43.291667 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:43.294398 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:43.298328 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.298337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.298343 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.298958 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.299411 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.299416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.299419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.299597 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.300296 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:43.300303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:43.300466 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 23:34:43.300665 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 23:34:43.301057 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":623,"threads_started":1}
I20260620 23:34:43.304924 23435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.254@KRBTEST.COM
I20260620 23:34:43.321524 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.305829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.253:41349 (local address 127.0.0.1:41300)
0620 23:34:43.306066 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.306090 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:43.306268 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:43.306875 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:43.306879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:43.307294 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:43.307474 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.307482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.308653 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.308659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.309810 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.309823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.310603 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.310609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.310790 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.311776 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:43.311796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:43.314667 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:43.318456 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.318467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.318473 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.319084 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.319541 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.319544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.319547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.319732 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.320317 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:43.320321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:43.320546 (+ 225us) client_negotiation.cc:769] Sending connection context
0620 23:34:43.320738 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 23:34:43.321147 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 23:34:43.321973 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.305897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.0.96.253:41349)
0620 23:34:43.306538 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.306545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.306567 (+ 22us) server_negotiation.cc:407] Connection header received
0620 23:34:43.306626 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.306632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.306709 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.306840 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.307678 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.308495 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.309975 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.310439 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.310537 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.314939 (+ 4402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.314949 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.314955 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.315003 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.318258 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.319212 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.319217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.319221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.319409 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.319857 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.319863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.319866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.320187 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.320347 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.320861 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.321146 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":344,"threads_started":1}
I20260620 23:34:43.322270 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.305296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.0.96.254:37163)
0620 23:34:43.305498 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.305502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.305658 (+ 156us) server_negotiation.cc:407] Connection header received
0620 23:34:43.305953 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.305960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.306064 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.306187 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.307836 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.308664 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.310048 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.310474 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.310641 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.315665 (+ 5024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.315671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.315674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.315718 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.319002 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.319908 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.319913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.319915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.320047 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.320469 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.320474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.320477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.320872 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.321244 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.321620 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.321909 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:34:43.323391 23435 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.252@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.96.253@KRBTEST.COM
I20260620 23:34:43.339403 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.324584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.0.96.252:37881)
0620 23:34:43.324770 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.324774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.324834 (+ 60us) server_negotiation.cc:407] Connection header received
0620 23:34:43.325053 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.325058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.325126 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.325235 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.326272 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.327044 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.328499 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.328948 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.329134 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.333269 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.333274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.333278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.333309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.336248 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.337274 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.337278 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.337281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.337413 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.337842 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.337845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.337847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.338183 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.338356 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.338727 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.338979 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 23:34:43.339520 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.324487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41792)
0620 23:34:43.324759 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.324778 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:43.324890 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:43.325275 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:43.325280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:43.325822 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:43.326060 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.326075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.327191 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.327196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.328332 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.328347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.329199 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.329206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.329453 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.330215 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:43.330235 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:43.333043 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:43.336481 (+ 3438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.336490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.336497 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.337134 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.337542 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.337549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.337553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.337722 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.338311 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:43.338316 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:43.338472 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 23:34:43.338641 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 23:34:43.339161 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":136}
I20260620 23:34:43.341194 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.324410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.0.96.253:41349)
0620 23:34:43.324578 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.324583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.324838 (+ 255us) server_negotiation.cc:407] Connection header received
0620 23:34:43.325032 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.325036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.325113 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.325228 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.327031 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.327777 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.329492 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.329881 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.330021 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.334918 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.334923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.334926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.334965 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.338077 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.338710 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.338717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.338725 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.338850 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.339228 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.339234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.339238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.339565 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.339865 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.340445 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.340772 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 23:34:43.345691 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: Bootstrap starting.
I20260620 23:34:43.347080 387 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 23:34:43.349269 23435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:43.350329 23435 log.cc:824] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:43.349948 23261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57312
I20260620 23:34:43.353080 23435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: No bootstrap required, opened a new log
I20260620 23:34:43.353813 23360 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 5
I20260620 23:34:43.363261 23435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.363797 23435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554489a6b6964aaa8869a29ad8e78c0a, State: Initialized, Role: FOLLOWER
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.253@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
I20260620 23:34:43.364351 23435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.365494 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.365934 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.366820 23435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:43.374898 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.354746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41798 (local address 127.0.96.252:37881)
0620 23:34:43.355016 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.355022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.355045 (+ 23us) server_negotiation.cc:407] Connection header received
0620 23:34:43.355272 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.355278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.355388 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.355527 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.356517 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.357934 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.359763 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.360600 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.360738 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.365728 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.365739 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.365745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.365807 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.370592 (+ 4785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.371748 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.371754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.371758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.371929 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.372776 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.372782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.372785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.373354 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.373544 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.374168 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.374466 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:34:43.375240 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.354652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41798)
0620 23:34:43.354901 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.354929 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:43.355069 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:43.355579 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:43.355584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:43.356100 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:43.356321 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.356344 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.358137 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.358143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.359575 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.359593 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.360784 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.360791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.361069 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.361995 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:43.362021 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:43.365387 (+ 3366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:43.370948 (+ 5561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.370956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.370961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.371548 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.372151 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.372163 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.372170 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.372578 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.373536 (+ 958us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:43.373545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:43.373798 (+ 253us) client_negotiation.cc:769] Sending connection context
0620 23:34:43.374020 (+ 222us) client_negotiation.cc:240] Negotiation successful
0620 23:34:43.374624 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260620 23:34:43.381940 23360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: Bootstrap starting.
W20260620 23:34:43.384902 23454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:34:43.385154 23454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:34:43.386158 23360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:43.387450 23360 log.cc:824] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:43.391500 23360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: No bootstrap required, opened a new log
I20260620 23:34:43.382680 23368 master_runner.cc:186] Error getting master registration for 127.0.96.254:37163: OK, instance_id { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" instance_seqno: 1781998482046715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:34:43.394400 23368 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:34:43.399552 23360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.399884 23360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f26e354e43414fa257d247e5f954f9, State: Initialized, Role: FOLLOWER
I20260620 23:34:43.400418 23360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.401417 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.401793 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.402568 23360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 23:34:43.422035 23468 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 23:34:43.422309 23468 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 23:34:43.411418 23293 master_runner.cc:186] Error getting master registration for 127.0.96.254:37163: OK, instance_id { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" instance_seqno: 1781998482046715 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 23:34:43.425726 23293 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 23:34:43.486722 23286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } attempt: 5
W20260620 23:34:43.494840 23389 tablet.cc:2369] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 23:34:43 dist-test-slave-v34r krb5kdc[23206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781998481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.96.254@KRBTEST.COM for kudu/127.0.96.252@KRBTEST.COM
I20260620 23:34:43.508268 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.488124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.0.96.252:37881)
0620 23:34:43.488381 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.488388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.488414 (+ 26us) server_negotiation.cc:407] Connection header received
0620 23:34:43.488490 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.488496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.488600 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.488763 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.489937 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.491152 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.493201 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.494000 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.494157 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.499893 (+ 5736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.499901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.499907 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.499968 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.504291 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.505587 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.505594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.505597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.505794 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.506464 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.506469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.506473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.506879 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.507053 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.507580 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.507854 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 23:34:43.508292 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.487576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.96.252:37881 (local address 127.0.0.1:41806)
0620 23:34:43.488105 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 23:34:43.488129 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 23:34:43.488287 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 23:34:43.488836 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 23:34:43.488843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 23:34:43.489502 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 23:34:43.489759 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.489774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.491352 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.491359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.492932 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 23:34:43.492962 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.494217 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.494224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 23:34:43.494481 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.495550 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 23:34:43.495580 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 23:34:43.499570 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 23:34:43.504568 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.504578 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.504585 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.505387 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.505957 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 23:34:43.505964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 23:34:43.505968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 23:34:43.506306 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 23:34:43.507030 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 23:34:43.507036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 23:34:43.507253 (+ 217us) client_negotiation.cc:769] Sending connection context
0620 23:34:43.507429 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 23:34:43.507801 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
I20260620 23:34:43.515470 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9: Bootstrap starting.
I20260620 23:34:43.519891 23286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9: Neither blocks nor log segments found. Creating new log.
I20260620 23:34:43.521397 23286 log.cc:824] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9: Log is configured to *not* fsync() on all Append() calls
I20260620 23:34:43.532181 23286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9: No bootstrap required, opened a new log
I20260620 23:34:43.540211 23286 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.540642 23286 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca7efb8ec6f45d7bd95492115fe8fc9, State: Initialized, Role: FOLLOWER
I20260620 23:34:43.541204 23286 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.557683 23471 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.558174 23471 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.559186 23286 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 23:34:43.559463 23261 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 23:34:43.559854 23261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:43.563987 23261 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.567477 23335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95f26e354e43414fa257d247e5f954f9"
I20260620 23:34:43.568094 23335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:43.572296 23335 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca7efb8ec6f45d7bd95492115fe8fc9 in term 1.
I20260620 23:34:43.573668 23237 leader_election.cc:302] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95f26e354e43414fa257d247e5f954f9, aca7efb8ec6f45d7bd95492115fe8fc9; no voters:
I20260620 23:34:43.574476 23471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 23:34:43.576140 23410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "554489a6b6964aaa8869a29ad8e78c0a"
I20260620 23:34:43.576807 23410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 23:34:43.578411 23471 raft_consensus.cc:686] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [term 1 LEADER]: Becoming Leader. State: Replica: aca7efb8ec6f45d7bd95492115fe8fc9, State: Running, Role: LEADER
I20260620 23:34:43.579155 23471 consensus_queue.cc:237] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } }
I20260620 23:34:43.580883 23410 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca7efb8ec6f45d7bd95492115fe8fc9 in term 1.
I20260620 23:34:43.582531 23261 leader_election.cc:288] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [CANDIDATE]: Term 1 election: Requested vote from peers 95f26e354e43414fa257d247e5f954f9 (127.0.96.253:41349), 554489a6b6964aaa8869a29ad8e78c0a (127.0.96.252:37881)
W20260620 23:34:43.596283 23314 tablet.cc:2369] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 23:34:43.599045 23473 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca7efb8ec6f45d7bd95492115fe8fc9. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.599546 23473 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:43.614037 23478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 23:34:43.624274 23478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 23:34:43.659173 23410 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [term 1 FOLLOWER]: Refusing update from remote peer aca7efb8ec6f45d7bd95492115fe8fc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:34:43.661588 23335 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [term 1 FOLLOWER]: Refusing update from remote peer aca7efb8ec6f45d7bd95492115fe8fc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 23:34:43.662946 23473 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:34:43.665179 23473 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 23:34:43.675254 23494 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 23:34:43.702678 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca7efb8ec6f45d7bd95492115fe8fc9. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.703083 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.720232 23490 mvcc.cc:205] Tried to move back new op lower bound from 7299065789002534912 to 7299065788747677696. Current Snapshot: MvccSnapshot[applied={T|T < 7299065789002534912}]
I20260620 23:34:43.725808 23443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.726958 23443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.742676 23471 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.743134 23471 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:43.752115 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.615291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.0.96.254:37163)
0620 23:34:43.615519 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.615527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.639388 (+ 23861us) server_negotiation.cc:407] Connection header received
0620 23:34:43.641734 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.641742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.641848 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.641926 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.651476 (+ 9550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.652798 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.686939 (+ 34141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.687694 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.687830 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.704062 (+ 16232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.704073 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.704078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.704164 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.708631 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.744551 (+ 35920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.744558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.744562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.744807 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.747961 (+ 3154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.747967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.747970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.748488 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.748835 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.751289 (+ 2454us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.751664 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 23:34:43.756748 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.621595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.0.96.253:41349)
0620 23:34:43.621809 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.621819 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.645096 (+ 23277us) server_negotiation.cc:407] Connection header received
0620 23:34:43.646251 (+ 1155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.646262 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.646373 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.646460 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.654952 (+ 8492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.656217 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.689710 (+ 33493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.690427 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.690572 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.710638 (+ 20066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.710648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.710654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.710738 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.715242 (+ 4504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.747530 (+ 32288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.747539 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.747544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.747775 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.749847 (+ 2072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.749854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.749858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.750326 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.750658 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.755940 (+ 5282us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.756271 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 23:34:43.760950 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:43.633843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.0.96.252:37881)
0620 23:34:43.634142 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.634149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.646090 (+ 11941us) server_negotiation.cc:407] Connection header received
0620 23:34:43.648930 (+ 2840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:43.648938 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:43.649078 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:43.649164 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:43.657460 (+ 8296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.658564 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.694747 (+ 36183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:43.695519 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:43.695648 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:43.716646 (+ 20998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 23:34:43.716655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 23:34:43.716662 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 23:34:43.716727 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0620 23:34:43.729726 (+ 12999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.754979 (+ 25253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.754986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.754990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.755222 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 23:34:43.758484 (+ 3262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 23:34:43.758491 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 23:34:43.758494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 23:34:43.759002 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 23:34:43.759351 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:43.760204 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 23:34:43.760528 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 23:34:43.765196 23478 catalog_manager.cc:1347] Generated new cluster ID: 9e5d3400f65d4f86aa9647585b2411b7
I20260620 23:34:43.765470 23478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 23:34:43.768661 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca7efb8ec6f45d7bd95492115fe8fc9. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.769121 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.778201 23473 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.778571 23473 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9 [sys.catalog]: This master's current role is: LEADER
I20260620 23:34:43.782613 23496 mvcc.cc:205] Tried to move back new op lower bound from 7299065789002534912 to 7299065788747677696. Current Snapshot: MvccSnapshot[applied={T|T < 7299065789002534912}]
I20260620 23:34:43.793752 23478 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 23:34:43.796429 23478 catalog_manager.cc:1504] Loading token signing keys...
I20260620 23:34:43.806448 23457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca7efb8ec6f45d7bd95492115fe8fc9" member_type: VOTER last_known_addr { host: "127.0.96.254" port: 37163 } } peers { permanent_uuid: "95f26e354e43414fa257d247e5f954f9" member_type: VOTER last_known_addr { host: "127.0.96.253" port: 41349 } } peers { permanent_uuid: "554489a6b6964aaa8869a29ad8e78c0a" member_type: VOTER last_known_addr { host: "127.0.96.252" port: 37881 } } }
I20260620 23:34:43.806869 23457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 23:34:43.832111 23478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aca7efb8ec6f45d7bd95492115fe8fc9: Generated new TSK 0
I20260620 23:34:43.833739 23478 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 23:34:44.003080 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:43.981127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.0.96.253:41349)
0620 23:34:43.981428 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.981435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.999978 (+ 18543us) server_negotiation.cc:407] Connection header received
0620 23:34:44.000966 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.001003 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.001134 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.001231 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.002693 (+ 1462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41330: BlockingRecv error: recv got EOF from 127.0.0.1:41330 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 23:34:44.005796 23439 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:43.973353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.0.96.254:37163)
0620 23:34:43.973671 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.973681 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:43.999368 (+ 25687us) server_negotiation.cc:407] Connection header received
0620 23:34:44.003680 (+ 4312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.003689 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.003821 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.003920 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.005452 (+ 1532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57334: BlockingRecv error: recv got EOF from 127.0.0.1:57334 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260620 23:34:44.011163 23438 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:43.993726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.0.96.252:37881)
0620 23:34:43.993987 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:43.993994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.006719 (+ 12725us) server_negotiation.cc:407] Connection header received
0620 23:34:44.008132 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.008142 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.008263 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.008370 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.010640 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41820: BlockingRecv error: recv got EOF from 127.0.0.1:41820 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260620 23:34:44.044284 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.039105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.0.96.253:41349)
0620 23:34:44.039366 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.039376 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.041966 (+ 2590us) server_negotiation.cc:407] Connection header received
0620 23:34:44.042275 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.042290 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.042444 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.042592 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.043943 (+ 1351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41344: BlockingRecv error: recv got EOF from 127.0.0.1:41344 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260620 23:34:44.052388 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.049540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.0.96.253:41349)
0620 23:34:44.049778 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.049785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.050472 (+ 687us) server_negotiation.cc:407] Connection header received
0620 23:34:44.050719 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.050726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.050852 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.050995 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.052066 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41356: BlockingRecv error: recv got EOF from 127.0.0.1:41356 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 23:34:44.091709 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.087160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.0.96.253:41349)
0620 23:34:44.087380 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.087387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.089771 (+ 2384us) server_negotiation.cc:407] Connection header received
0620 23:34:44.090006 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.090015 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.090147 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.090308 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.091329 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41362: BlockingRecv error: recv got EOF from 127.0.0.1:41362 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 23:34:44.111712 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.106044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.0.96.253:41349)
0620 23:34:44.106353 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.106361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.109884 (+ 3523us) server_negotiation.cc:407] Connection header received
0620 23:34:44.110127 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.110135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.110263 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.110417 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.111414 (+ 997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41376: BlockingRecv error: recv got EOF from 127.0.0.1:41376 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
W20260620 23:34:44.126448 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.123824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.0.96.253:41349)
0620 23:34:44.124103 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.124111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.124536 (+ 425us) server_negotiation.cc:407] Connection header received
0620 23:34:44.124800 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.124807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.124949 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.125121 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.126083 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41380: BlockingRecv error: recv got EOF from 127.0.0.1:41380 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260620 23:34:44.149736 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.146738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41394 (local address 127.0.96.253:41349)
0620 23:34:44.146980 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.146986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.147699 (+ 713us) server_negotiation.cc:407] Connection header received
0620 23:34:44.148016 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.148022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.148129 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.148261 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.149411 (+ 1150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41394: BlockingRecv error: recv got EOF from 127.0.0.1:41394 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260620 23:34:44.188732 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.186291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.96.253:41349)
0620 23:34:44.186601 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.186609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.187035 (+ 426us) server_negotiation.cc:407] Connection header received
0620 23:34:44.187316 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.187321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.187438 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.187538 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.188447 (+ 909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41400: BlockingRecv error: recv got EOF from 127.0.0.1:41400 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260620 23:34:44.259572 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.257322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.0.96.253:41349)
0620 23:34:44.257637 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.257645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.257917 (+ 272us) server_negotiation.cc:407] Connection header received
0620 23:34:44.258117 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.258122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.258219 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.258340 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.259233 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41408: BlockingRecv error: recv got EOF from 127.0.0.1:41408 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 23:34:44.387337 23454 catalog_manager.cc:1259] Loaded cluster ID: 9e5d3400f65d4f86aa9647585b2411b7
I20260620 23:34:44.387454 23454 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: loading cluster ID for follower catalog manager: success
I20260620 23:34:44.389989 23454 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: acquiring CA information for follower catalog manager: success
I20260620 23:34:44.392117 23454 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 554489a6b6964aaa8869a29ad8e78c0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 23:34:44.394405 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 23:34:44.392315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.0.96.253:41349)
0620 23:34:44.392606 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.392614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.392930 (+ 316us) server_negotiation.cc:407] Connection header received
0620 23:34:44.393192 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.393197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.393277 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.393393 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 23:34:44.394128 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41422: BlockingRecv error: recv got EOF from 127.0.0.1:41422 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260620 23:34:44.424952 23468 catalog_manager.cc:1259] Loaded cluster ID: 9e5d3400f65d4f86aa9647585b2411b7
I20260620 23:34:44.425154 23468 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: loading cluster ID for follower catalog manager: success
I20260620 23:34:44.428138 23468 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: acquiring CA information for follower catalog manager: success
I20260620 23:34:44.430998 23468 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95f26e354e43414fa257d247e5f954f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 23:34:44.665446 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:44.658902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41436 (local address 127.0.96.253:41349)
0620 23:34:44.659203 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.659209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.659493 (+ 284us) server_negotiation.cc:407] Connection header received
0620 23:34:44.659824 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.659830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.660025 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.660160 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:44.660630 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.661456 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.662843 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.663243 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.663515 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:44.664064 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:44.664780 (+ 716us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:44.664941 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:44.665063 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 23:34:44.665158 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260620 23:34:44.680732 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:44.673411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41836 (local address 127.0.96.252:37881)
0620 23:34:44.674055 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.674060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.674181 (+ 121us) server_negotiation.cc:407] Connection header received
0620 23:34:44.674658 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.674663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.674857 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.675034 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:44.675562 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.676341 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.678122 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.678535 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.678642 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:44.679230 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:44.679851 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:44.679974 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:44.680180 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 23:34:44.680267 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":372,"threads_started":1}
I20260620 23:34:44.701613 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:44.688909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.0.96.253:41349)
0620 23:34:44.689444 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.689454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.694298 (+ 4844us) server_negotiation.cc:407] Connection header received
0620 23:34:44.694597 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.694603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.694698 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.694821 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:44.695735 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.696958 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.699303 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.700028 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.700162 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:44.700748 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:44.700879 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:44.700971 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:44.701188 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 23:34:44.701299 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260620 23:34:44.701860 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:44.689253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.0.96.254:37163)
0620 23:34:44.692910 (+ 3657us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.692918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.693656 (+ 738us) server_negotiation.cc:407] Connection header received
0620 23:34:44.694137 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.694144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.694427 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.694685 (+ 258us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:44.695408 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.696269 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.698532 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.699273 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.699449 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:44.700188 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:44.700587 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:44.700715 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:44.701444 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 23:34:44.701546 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3482,"thread_start_us":3324,"threads_started":1}
I20260620 23:34:44.702275 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 23:34:44.688885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41846 (local address 127.0.96.252:37881)
0620 23:34:44.689169 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 23:34:44.689178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 23:34:44.694012 (+ 4834us) server_negotiation.cc:407] Connection header received
0620 23:34:44.694334 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 23:34:44.694338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 23:34:44.694410 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 23:34:44.694512 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 23:34:44.695689 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.696958 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.699945 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 23:34:44.700641 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 23:34:44.700782 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 23:34:44.701348 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 23:34:44.701512 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 23:34:44.701586 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 23:34:44.701835 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 23:34:44.701931 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260620 23:34:44.709288 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 23219
I20260620 23:34:44.710704 23281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:44.841241 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 23219
W20260620 23:34:44.863592 23309 connection.cc:535] server connection from 127.0.0.1:41296 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 23:34:44.864113 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 23293
I20260620 23:34:44.865584 23355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:44.979491 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 23293
I20260620 23:34:45.004642 387 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu pid 23368
I20260620 23:34:45.006160 23430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 23:34:45.120455 387 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWbSr8K/build/asan/bin/kudu with pid 23368
2026-06-20T23:34:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5034 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10212 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (397329 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST